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Foreword 


The National Forum on Education Statistics (Forum) is pleased to present the Forum Guide to Early Warning Systems. The 
purpose of this document is to provide information and best practices that will help education agencies plan, develop, 


implement, and use an early warning system in their agency to inform interventions that improve student outcomes. 


This document focuses on early warning systems and their data from the perspective of the education data community. It 
does not attempt to present a comprehensive account of the many issues related to the design, implementation, and use 


of early warning systems in education agencies. 


Publication Objectives 


This resource is intended to help state and local education agencies plan, develop, implement, and use an early warning 


system in their agency. It strives to 


* provide background information on early warning systems and their use in education agencies; 

* — illustrate the ways in which these systems can help education agencies act on early warning data to improve 
student outcomes; 

* explain the essential role of timely and actionable quality data in early warning systems; and 

* recommend strategies and best practices for education agencies that are considering implementing, or have 


already implemented, an early warning system. 


Intended Audience 


The Forum Guide to Early Warning Systems is intended for staff in education agencies that are considering implementing, or 
have already implemented, an early warning system. This includes program and data staff, administrators, policymakers, 
and anyone else involved in planning, developing, implementing, or using an early warning system at the school, district, 
or state level. This resource may also be relevant to stakeholders with an interest in supporting improved student 


outcomes, including students, parents, community agencies, and other parties. 


Organization of This Resource 


This resource includes the following chapters and appendices: 


* Chapter 1 defines early warning systems, reviews their use in education agencies, and explains how to adopt 
an effective early warning system planning process. 

* Chapter 2 reviews the role of indicators and quality data in early warning systems, summarizes techniques 
for analyzing early warning data, and previews trends and innovations in early warning systems, indicators, 
data, and analytical models. 

* Chapter 3 recommends best practices in developing, implementing, and using early warning systems. 

* Chapter 4 highlights case studies from state and local education agencies that have implemented, or are in the 
process of implementing, an early warning system. 

* Appendix A: Checklist of Early Warning System Tasks and Activities contains a checklist of tasks 
and activities involved in planning, implementing, using, and evaluating an early warning system. 

* Appendix B: Select List of Early Warning Indicators and Data Elements lists early warning 


indicator topics, student-level indicators, and related data elements. 


National Forum on Education Statistics 


The work of the National Forum on Education Statistics (Forum) is a key aspect of the National Cooperative Education 
Statistics System (Cooperative System). The Cooperative System was established to produce and maintain, with the 
cooperation of the states, comparable and uniform education information and data that are useful for policymaking at 
the federal, state, and local levels. To assist in meeting this goal, the National Center for Education Statistics (NCES) 
within the Institute of Education Sciences (IES)—a part of the U.S. Department of Education (ED)—established the 
Forum to improve the collection, reporting, and use of elementary and secondary education statistics. The Forum 
includes approximately 120 representatives from state and local education agencies, the federal government, and 

other organizations with an interest in education data. The Forum deals with issues in education data policy, sponsors 


innovations in data collection and reporting, and provides technical assistance to improve state and local data systems. 


Development of Forum Products 


Members of the Forum establish working groups to develop guides in data-related areas of interest to federal, state, and 
local education agencies. They are assisted in this work by NCES, but the content comes from the collective experience 
of working group members who review all products iteratively throughout the development process. After the working 
group completes the content and reviews a document a final time, publications are subject to examination by members 
of the Forum standing committee that sponsors the project. Finally, Forum members review and formally vote to 
approve all documents prior to publication. NCES provides final review and approval prior to online publication. The 
information and opinions published in Forum products do not necessarily represent the policies or views of ED, IES, or 


NCES. Readers may modify, customize, or reproduce any or all parts of this document. 
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Glossary of Common Terms 


ABCs — Attendance, behavior, and course performance. 
Data Element — The most basic unit of specific information that can be defined, measured, and stored. 
Dropout — Withdrawal from an educational program before completion. 


Dropout Prevention — Intervention programs designed to keep students in school and ultimately improve their 
likelihood of completing high school. 


Early Warning System — A system that identifies students who are at risk of dropping out of school. This type of 
system applies predictive analytics to student data to determine student risk level in relation to predefined indicators and 
thresholds. By identifying students who are at-risk, an early warning system provides targeted data that may be used to 


develop student interventions that are designed to avoid negative student outcomes. 


Event Dropout Rate — The percentage of high school students who left school between the beginning of one school 


year and the beginning of the next without earning a high school diploma or an alternative credential. 


General Educational Development (GED®) Test — A standardized test designed to measure the skills and 
knowledge that students normally acquire by the end of high school. The test is developed by the American Council on 


Education’s GED Testing Service. People who pass may receive an alternative high school credential. 


HiSET Exam — A standardized test designed to measure the skills and knowledge that students normally acquire by the 
end of high school. The test is developed by the Educational Testing Service. People who pass may receive an alternative 
high school credential. 


Indicator — A measure of the status of, or change in, an entity with regard to its goals. In the context of an early 
warning system, early warning indicators measure the extent to which a student is likely to be at risk of dropping out of 


school. 
Intervention — An educational program, product, practice, or policy aimed at positively influencing student outcomes. 


Intervention System — A system that uses multiple methods to determine which students will receive interventions, 
and to what degree, and administers interventions to students through a defined process. This type of system may or may 


not be informed by an early warning system. 


Longitudinal Data System — A data system that collects and maintains detailed, high-quality, student- and staff-level 
data that are linked across entities and over time, providing a complete academic and performance history for each 


student, and makes these data accessible through reporting and analysis tools. 
Multi-Tiered System of Supports (MTSS) — A comprehensive tiered intervention model that aims to improve 


both the academic and behavioral performance of all students within a school. This type of system may or may not be 


informed by an early warning system. 
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Positive Behavioral Interventions and Supports (PBIS) — A tiered intervention model that aims to reduce and 
prevent problem behaviors and improve the climate, discipline, and behavioral performance of all students within a 


school. This type of system may or may not be informed by an early warning system. 


Predictive Analytics — Statistical techniques that use a combination of data-based indicators, tests, or measures to 


determine or estimate future events, conditions, outcomes, or trends. 


Response to Intervention (RTI) — A tiered intervention model that aims to improve the academic performance of 


all students within a school. This type of system may or may not be informed by an early warning system. 


Risk Factor — A condition or variable that may be predictive of a student dropping out. Risk factors traditionally 
include attendance, behavior, and course performance, and may include other factors such as exhibited behaviors or 


academic performance. 


Status Completion Rate — Among 18- to 24-year-olds who are not enrolled in high school or a lower education level, 
the percentage who hold a high school diploma or alternative credential (such as a GED). 


Status Dropout Rate — The percentage of individuals in a given age range (typically 16- to 24-year-olds) who are not 


enrolled in school and have not earned a high school diploma or alternative credential. 


System — A set of regularly interacting parts that interact, or are assembled, to form a unified whole and serve a 


common purpose. 
Test Assessing Secondary Completion™ (TASC) — A standardized test designed to measure the skills and 
knowledge that students normally acquire by the end of high school. The test is developed by the Data Recognition 


Corporation. People who pass may receive an alternative high school credential. 


Tier I Intervention — A universal, schoolwide intervention that is provided to all students (Regional Educational 
Laboratory Pacific 2016). 


Tier II Intervention — A targeted intervention that is provided to a select group of students in need of additional 
intervention, typically 15-20 percent of the student population (Regional Educational Laboratory Pacific 2016). 


Tier III Intervention — An intensive intervention that is provided to a small group of students who are in greatest 
need of intervention, typically 5-10 percent of the student population (Regional Educational Laboratory Pacific 2016). 
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Chapter 1: 


Introduction to Early Warning Systems 


High school graduation rates in the United States have shown steady increases in recent years, reaching a record high in 
2014 of 92.4 percent among 18- to 24-year-olds (McFarland, Cui, and Stark 2018). During this same period, over half 

a million students ages 15-24 left school without obtaining a high school diploma or earning a high school equivalency 
credential by passing a comprehensive equivalency examination, such as the General Educational Development (GED®) 
test, Test Assessing Secondary Completion™ (TASC), or HiSET exam. While high school graduation rates have improved 


over time, the need to support students who are at risk of dropping out of school remains. 


There are many important reasons to remain focused on supporting Trends in High School Dropout and 

student progression, achievement, and graduation. Timely academic Completion Rates in the United States: 2014 
https://nces.ed.gov/pubsearch/pubsinfo. 
asp?pubid=2018117 

This National Center for Education Statistics 
school diploma, those with at least a high school diploma were more (NCES) report draws on a wide variety of 


progression and graduation can strongly influence a student’s future 


life outcomes. When compared with individuals with less than a high 


likely to be employed and to have higher median weekly earnings data sources to examine high school dropout 
(U.S. Department of Labor 2017). Individuals who drop out of and completion rates, as well as data on 
long-term trends. This report describes the 
characteristics of high school dropouts and 


completers, including race/ethnicity, sex, 
Ward, and Lucas 2010). Graduating from high school can improve socioeconomic status, disability status, 


high school are also at risk of poorer health and a shorter lifespan 


in comparison to those who have graduated (Rouse 2007; Pleis, 


individual outcomes and provide graduates with opportunities that recency of immigration, and outcomes in the 


positively impact their lives. labor force. 


In recognition of the critical need to support educational attainment and graduation, many state and local education 
agencies (SEAs and LEAs) are considering methods and strategies to improve the provision of student interventions, 
services, and supports. As of the 2014-15 school year, about half of public high schools reported using an early warning 
system to identify students who are at risk of dropping out of school (U.S. Department of Education 2016). Early 
warning systems strategically use data to identify students in need of additional support and interventions that will help 
improve student success. By identifying students as soon as they begin to demonstrate some risk of not completing high 
school, early warning systems may support the provision of early interventions, which are typically more effective and 


affordable than late interventions. 


Early Warning Systems 


An early warning system identifies students who are at risk of dropping out of school. This type of system applies 
predictive analytics to student data to determine student risk level in relation to predefined indicators and thresholds. 


Risk factors traditionally include attendance, behavior, and course performance, and may include other factors such as 
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exhibited behaviors or academic performance. By identifying students who are at-risk, an early warning system provides 
targeted data that may be used to develop student inter ventions that are designed to avoid negative student outcomes. 
Early warning systems are more than just a technology system—they are a tool in the process of identifying and 


supporting students who are at risk of not achieving their educational goals. 


System Scope 


The scope of an early warning system is influenced by an education agency’s priorities, resources, and policies. Early 
warning systems may focus on students who are at imminent risk of dropping out, students who are at future risk of 
dropping out, or both. Agencies may be constrained in whether they can focus on students who are at imminent or lesser 
and/or future risk. Ideally, an agency would have staff teams who can address the needs of, and design interventions for, 


both populations. 


In addition to identifying students who are at risk of dropping out of school, early warning systems may also identify 
student risk levels in relation to other outcomes, such as on-time graduation, grade-level assessment thresholds, or 
specific academic achievement criteria, such as college and career readiness. A student may be considered at risk if, for 
example, he or she fails a fourth grade reading level assessment, experiences significant declines in grade point average 
(GPA), or performs poorly on a state graduation exam. 


Early warning systems may be used differently at various grade levels. High school early warning systems may assess 
whether students are at risk of dropping out before the end of the school year, while elementary school systems may 
instead assess whether students are exhibiting risk factors that indicate they may struggle in high school. Thus, an 
effective intervention for an elementary school student who is not performing at grade level would be different than an 


intervention for a high school student who is at risk of dropping out. 


While early warning systems identify students who are at-risk, they are not a diagnostic method for identifying the root 
cause of why a student is exhibiting risk. Further investigation is needed to understand the root causes of student risk 
and provide targeted interventions that are tailored to each student’s unique needs and address the root causes of the 


risk. 


Related Systems 


Early warning systems provide information on the level of risk exhibited by students, with the intention that students 
who exhibit the highest risk of dropping out will receive targeted interventions. Early warning systems are designed 

to inform student interventions and can support related systems, such as intervention systems, multi-tiered systems of 
supports (MTSS), and response to intervention (RTI). While some early warning systems work in tandem with these 
related systems, some do not. For example, a statewide early warning system that provides information on student risk 
levels would likely not include an intervention system, as SEAs are not typically responsible for implementing local 
student interventions. However, LEAs can use a statewide early warning system as a source of information to support 
their implementation of an intervention system at the school or district level. 


System Outcomes 


Supporting student achievement and success is an essential component of the responsibilities of education agencies at the 


school, district, state, and federal levels. Early warning systems enable schools, districts, and states to make informed 
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decisions and direct resources and services to those students who are in need of intervention. SEAs and LEAs can use an 
early warning system to identify which students are at-risk, determine the extent to which each student is at-risk, and 
decide whether to initiate a root-cause analysis to determine why each student may be at risk. Accurate, secure, useful, 
and timely data about the at-risk status of students helps education agencies provide intervention programs and services 
that support student needs and address the root causes of student risk. This critical insight into student risk provides 
SEAs and LEAs with actionable information that can guide the development and provision of targeted interventions to 


reduce dropout risk and promote student progression, achievement, and graduation. 


The provision of adequate and appropriate intervention programs and services can improve student outcomes and 
promote the achievement and success of students who are at risk of dropping out. Successfully using these systems 
to prevent students from dropping out helps ensure continued system use. Reducing dropout rates may also generate 


increases to total per-pupil funding, which could, in certain cases, make these systems a good return on investment. 


Resource Spotlight: Early Warning Systems 


Several agencies and organizations provide resources and technical assistance on early warning systems. SEAs and LEAs 
may find it useful to consult these resources as they consider developing a new, or refining an existing, early warning 
system. 


Development and Use of Early Warning Systems: SLDS Spotlight 
https://nces.grads360.org/#communities/slds-best-practices-conference/publications/2738 

This brief was derived from a Statewide Longitudinal Data Systems (SLDS) Grant Program webinar that focused on the 
development and use of state-level early warning systems in Maine, Massachusetts, and South Carolina. Presenters from 
each state discussed the benefits and challenges of implementing an early warning system and shared best practices for 
stakeholder input, planning, validation, training, and data use. 


Early Warning Indicators and Segmentation Analysis: A Technical Guide on Data Studies That Inform Dropout Prevention 
and Recovery 

https://files.eric.ed.gov/fulltext/ED577226.pdf 

This technical resource walks districts and schools through the early warning system planning, development, and 
implementation process. The guide details the specific technical steps involved in establishing an early warning system 
and conducting a segmentation study. 


Early Warning Systems in Education 

http://www.earlywarningsystems.org_ 

American Institutes for Research (AIR) has developed action-oriented, research-based resources and tools that states, 
districts, and schools can use to identify students in need of targeted interventions and supports to help student 
achievement and success. This website offers tools, resources, technical assistance, and training to support the design 
and implementation of early warning systems. 


Early Warning Systems Learning Series 

httos://ies.ed.gov/ncee/edlabs/projects/ews.asp 

The Institute of Education Sciences’ Regional Educational Laboratory (REL) Program has developed a series of projects 
on implementing and using early warning systems for dropout prevention. Resources related to projects in this series 
include webinar recordings, in-person event materials, videos, tools, infographics, reports, and works in progress. Useful 
resources include A Practitioner’s Guide to Implementing Early Warning Systems (Frazelle and Nagel 2015), Getting 
Students on Track for Graduation: Impacts of the Early Warning Intervention and Monitoring System After One Year (Faria 
et al. 2017), and Four Signs Your District is Ready for an Early Warning System: A Discussion Guide (REL Pacific 2016), 
amongst many more. 
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Early Warning System Planning Process 


Early warning systems require time, money, and effort. A continuous improvement process (figure 1) can help 

ensure that an early warning system is designed to support agency needs; is used appropriately and effectively in 
decisionmaking; and is responsive to changes in student populations, data practices, and agency processes. The tasks and 
activities in this process should be revisited regularly to assess the ongoing relevance and effectiveness of the system, as 


well as to identify opportunities for improvement. 


PLAN 


¢ Form a planning team 

* Conduct planning activities 
¢ Identify indicators and data 
¢ Develop an analytical model 


EVALUATE —— IMPLEMENT —— 


¢ Evaluate system accuracy * Allocate resources 
and usefulness ° Develop/refine an early 


¢ Document stakeholder feedback warning system 


¢ Determine needed improvements ¢ Train staff 
° Collect/import data 


¢ Analyze and interpret data 


USE 


¢ Report and share data 


* Provide intervention programs 
and services 


¢ Monitor student progress 


Figure 1. Continuous improvement process for early warning systems 


A list of tasks related to planning, implementing, using, and evaluating an early warning system is included in appendix 
A. Education agencies may find it useful to align the tasks and activities in this process with their organization’s needs 
and operational processes. Consulting resources on early warning systems (such as those listed in the Resource Spotlight 
textboxes in this guide) during the initial stages of the planning process can help agencies develop a robust approach to 


system planning and implementation. 


Form a Planning Team 


The development, implementation, and use of an early warning system require technical and analytical expertise, a 
broad-based commitment to data use, and an understanding of the agency and its resources. An early warning system 


planning team can provide the project management oversight necessary for system success. 
Early warning system planning, development, implementation, and use involves all aspects of an education agency. A 


representative planning team can strengthen the planning process and ensure that an early warning system is feasible 


and sustainable and meets agency and stakeholder needs. An early warning system created without the involvement 
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of representative staff and agency stakeholders might not 


An early warning system planning team may include 


incorporate key information necessary to support agency 
agency administrators and leadership; 
policy and research staff; 


data management and technical operations staff; ; ; . . 
student intervention and support staff: and technical operations staff, student intervention and 


needs. Agency representatives may include administrators 


and leadership, policy and research staff, data management 


teachers: support staff, teachers, and financial decisionmakers. 
financial decisionmakers; and 
representatives from other stakeholder groups. A diverse and representative planning team can contribute 


valuable expertise from the perspective of each member’s 


department or program area. However, different parts of 
the agency may have different goals and objectives for an early warning system. As a first step in the planning process, 
planning teams can work together to define the purpose of an early warning system and develop goals and objectives 
for system planning, implementation, and use. This can encourage the planning team to develop consensus and ensure 
that all members share a common understanding—and contribute towards the advancement—of the system planning 
process. 


Planning Activities 


With a planning team in place, the agency can begin creating a plan for implementing and using an early warning system 
to identify students who are at-risk. Team contributions can be enhanced through clearly defined roles and assigned 


responsibilities for individual team members. 
The planning process may include various activities, such as 


* conducting a needs assessment to identify agency and stakeholder needs; 

* engaging stakeholders through focus groups, interviews, or public presentations and meetings; 

* identifying and validating early warning indicators, data elements, and analytical models; 

* evaluating whether a system should be developed in-house by agency staff or procured from an external 
vendor or organization; 

* creating a system development plan or request for proposal that includes goals, timelines, and deliverables; 

* determining how end users will access and use the data, and developing a plan to ensure end users are 
involved in the system design process; and 

* developing a budget that includes a cost-benefit analysis and a projection of staff time necessary for system 


development, implementation, and maintenance. 
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Questions for Consideration 


When forming an early warning system planning team, it is helpful to consider who will be represented on the team, which 
activities the team will undertake, and how team members will be expected to contribute to the planning process. 


What factors are motivating your agency to consider an early warning system? 
Will the early warning system be created as part of a program, grant, or agency initiative? 
What types of planning activities will your team undertake before developing the early warning system? 
Conduct a needs analysis. 
Engage stakeholders. 
Identify and validate indicators, elements, and analytical models. 
Evaluate in-house development or vendor purchase options. 
Create a system development plan. 
Plan for end-user access. 
Develop a budget. 
Which staff role(s) will be involved in the early warning system planning process? 
Administrators 
Teachers 
Research staff 
Data management staff 
Student intervention and support staff 
Financial decisionmakers 
Will agency staff create the system, or will your agency work with an external group (e.g., a vendor, consultant, or 
other education agency) to create the early warning system? 
2 If agency staff will create your system, which departments and staff will be responsible for developing, 
implementing, and maintaining the system? 
If an external vendor or organization will create your system, which departments and staff will be responsible for 
overseeing procurement, implementation, and maintenance? 
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Early warning indicators, data, and analytics enable early warning systems to generate the information needed to identify 
students who are at risk of dropping out of school. Defining valid early warning indicators, collecting high-quality data, 
and identifying appropriate analytical techniques prior to system development can ensure that a system is aligned with 


agency needs, resources, and processes. 


Early Warning Indicators 


An education indicator is a measure of the status of, or change in, FOR Chino To Ednatoninticnters 


an educational system with regard to its goals (National Forum ’ ces.ed.gov/forum 200 
on Education Statistics 2005). In the context of an early warning 
system, early warning indicators measure the extent to which a 


student is likely to be at risk of dropping out of school. 


In determining which indicators to use in an early warning system, it 


is helpful to consider the following: 


¢ Have research studies found that the indicator is an 
effective predictor of dropout risk? 

* — Has the indicator passed local validation tests? 

* Are the data needed for the indicator currently collected 


and readily available, or might it be feasible to implement 


a new data collection? 
* Are the data used in calculating the indicator of high 
quality? 


* Does the indicator improve overall system accuracy? 


Identify and Define Indicators 


To be most effective, indicators and thresholds should be locally validated and applicable to the student population of 
that school, district, or state, as what is considered an at-risk threshold for students in one locality may not be the same 
in another locality. Indicator identification may occur through an analysis of preexisting internal longitudinal data that 
seeks to identify predictive trends and patterns, a review of external research conducted by other education agencies and 


organizations, or a combination of both internal analysis and external consultation. 
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State and local education agency (SEA and LEA) planning teams need to determine whether nationally recognized 
indicators are effective within the context of their school, district, or state. Specific research-based early warning 
indicators, such as the ABC indicators (attendance, behavior, and course performance) are well established and have been 
verified through research studies (e.g., Allensworth and Easton 2007; Balfanz, Herzog, and Mac Iver 2007; Mac Iver 
2010). 


A backward design process, which involves determining outcomes before identifying potential indicators, can be a useful 
approach for developing early warning indicators. Once the planning team determines which outcomes stakeholders 
would like their students to attain, student data can be segmented and analyzed based on whether students did, or 

did not, meet those outcomes to determine what, if any, differences exist between these two groups. The results of a 
segmentation analysis can guide the development of locally valid early warning indicators and thresholds (Jobs for the 
Future 2014). 


Indicator development is most effective when guided by research and data that demonstrate the appropriateness 

and accuracy of the indicator’s prediction of dropout risk. Determining indicator effectiveness based on individual 
perceptions may unintentionally bias the accuracy of a system. For example, tardiness may be predictive of whether a 
student will drop out of school, but absence data may be of higher quality and more effective at predicting dropout risk. 
Research staff can assist planning teams with indicator identification and validation. If an SEA or LEA does not have 
researchers on staff, external partners such as academic researchers, Regional Educational Laboratory (REL) program 


staff, or data consultants could work with the agency to identify and validate indicators. 


Commonly Used Early Warning Indicators 


Early warning systems commonly include indicators related to the ABCs: attendance, behavior, and course performance 
(U.S. Department of Education 2016). Some early warning systems, particularly at the state level, include assessment 
performance indicators and mobility indicators. General student data, such as identification and enrollment data, are not 


commonly used as indicators but may be included in an early warning system. 


Table 1 includes a list of commonly used early warning indicator areas that SEAs and LEAs have used in their early 
warning systems. The list of indicators is not exhaustive, nor is the accurate identification of student risk contingent on 
the use of these indicators. SEAs and LEAs may wish to consult the case studies in chapter 4 to learn more about various 
agencies’ early warning systems. A select list of early warning indicator topics, student-level indicators, and related data 


elements is included in appendix B. 
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OXoyenvastoyal bavelor-nvoyucy 

Assessment performance 

Achievement or performance on a school, district, and/or 
state assessment, typically a test 

Attendance 

Attendance in an instructional program approved by the 
school, district, and/or state! 


Behavior 
Negative or inappropriate behavior that results in a 
disciplinary action 


Course performance 


Academic achievement or performance across all courses 
ot a select set of courses 


Mobility 
Geographic movement between schools, districts, and/or 
states 


Progression 
On-time advancement to the next level or grade in an 
instructional program 


SEA and LEA Case Studies (Chapter 4) 
Appoquinimink School District, DE 

Fairbanks North Star Borough School District, AK 
Wisconsin Department of Public Instruction 
Appoquinimink School District, DE 

Bozeman School District #7, MT 

Delaware Department of Education 

Fairbanks North Star Borough School District, AK 
Metro Nashville Public Schools, TIN 

Montana Office of Public Instruction 

Wisconsin Department of Public Instruction 
Appoquinimink School District, DE 

Bozeman School District #7, MT 

Delaware Department of Education 

Fairbanks North Star Borough School District, AK 
Metro Nashville Public Schools, TIN 

Montana Office of Public Instruction 

Wisconsin Department of Public Instruction 
Appoquinimink School District, DE 

Bozeman School District #7, MT 

Delaware Department of Education 

Fairbanks North Star Borough School District, AK 
Metro Nashville Public Schools, TN 

Montana Office of Public Instruction 
Appoquinimink School District, DE 

Bozeman School District #7, MT 

Montana Office of Public Instruction 

Wisconsin Department of Public Instruction 
Appoquinimink School District, DE 

Bozeman School District #7, MT 

Fairbanks North Star Borough School District, AK 
Montana Office of Public Instruction 

Wisconsin Department of Public Instruction 


' For more information, see the Forum Guide to Collecting and Using Attendance Data at https:/ /nces.ed.gov[forum[ pub _2017007.asp. 


Table 1. Commonly used early warning indicators and agencies that use these indicators 
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Early Warning System Data 


The use of a given indicator is predicated on whether the related data are available, valid, and reliable. Data availability is 
particularly influential in developing statewide early warning indicators since SEAs have fewer student-level data points 
than LEAs. Reviewing preexisting data collections and mapping data elements to related indicators can help agencies 


determine whether high-quality data are available for each proposed indicator. 
In determining which data to use in an early warning system, it is helpful to consider the following: 


* How many years of data will be used? 

¢ = Which data are available? 

e Are the data recent? 

* Are the data timely? 

* Are the data collected regularly to support system predictability? 
* What is the quality of the data? 


Develop a Culture of Data Quality 


The effectiveness of an early warning system is contingent on the Forum Guide to Building a Culture of Quality 
quality of the data. Quality data are accurate, secure, useful, and Data: A School and District Resource 

timely (National Forum on Education Statistics 2004). A thorough 
understanding of the nature of early warning data, agency data 
collection processes, and system data analysis techniques is essential 


for validating data quality. 


The timeliness of data and, by extension, the frequency of data 
collection, reporting, and analysis processes is an essential 
component of system data quality. The definition of “timely data” 
may differ across agencies. A statewide, SEA-operated early warning 
system may use the most recent year’s annual summative data, while 


a districtwide, LEA-operated system may be dynamic and update 


data daily to identify students more readily. This difference impacts 
the types of action an agency can take based on the data. 


When establishing a schedule for data updates, agencies also need to consider the purpose and goals of the system. 
Frequently updated data may be more accurate in identifying risk status if it is additive; that is, if it is retained by the 
system and contributes to the system’s predictive abilities. Ongoing, systematic data processes enable early warning 
systems to identify students in need of intervention as soon as they meet a preexisting threshold. Agencies that update 
their early warning system data frequently and consistently can rapidly identify students who meet thresholds and 

design interventions for those students, while agencies that update their data only once or twice a year may miss key 
opportunities to support students in need of interventions. Systems that assess student risk using frequently updated data 
may be most useful for identifying older students who are at imminent risk of dropping out of high school, whereas a 
system that identifies elementary school-age students that demonstrate future risk of dropping out may not require real- 


time data. 
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Protect Data Privacy and Security 


As with any system that contains personally identifiable information, 


Forum Guide to Education Data Privacy 


care must be taken to ensure that student data are private and 
secure. SEAs and LEAs must ensure that early warning system data 
are protected, secure, and used only for the intended purpose of the 
system. Numerous laws, including the Family Educational Rights 
and Privacy Act (FERPA) (20 U.S.C. § 1232g; 34 CFR Part 99), 
have been enacted at the federal, state, and local levels to protect 
student information and data. Guidance and legally authoritative 
resources on protecting student privacy, including publications, 
training materials, and technical assistance, are available through the 
U.S. Department of Education’s Office of the Chief Privacy Officer 
website: https: //studentprivacy. ed. gov. 


Collect Accurate and Timely Data 


Data used in early warning systems are often already available as part of other data collections that occur in the school, 
district, or state. For statewide early warning systems, SEAs may be able to use data already reported by LEAs. For 
district- and school-level early warning systems, LEAs may be able to use data already collected at the local level or draw 
on data used in a statewide early warning system. For example, the Appoquinimink School District (DE) uses regularly 
collected data to update its early warning system each marking period, and thus did not need to implement any new data 


collections to support the system. 


While some data already exist and remain relatively constant over time, other data are generated through one-time or 
recurring events and activities. Existing data need to be mapped to or aligned with the early warning system before 
being imported or uploaded, and new data will need to be collected as students progress through school. For example, 
student identification information often remains constant, grade-level assessment scores may be generated one time, 

and attendance status data are generated each school calendar day or, in some cases, every class period, depending on 
how frequently attendance is recorded. In the Irving Independent School District in Texas, historical student attendance 
data are used to identify students whose attendance will be monitored daily by an attendance administrator charged with 
intervening to keep the student in school. The Forum Guide to Collecting and Using Attendance Data (2018) provides best 


practice information on the collection, reporting, and use of attendance data in education agencies. 


If early warning data are missing, nonstandard, or not regularly collected, it is important to consider the burden and 
costs of implementing new data collections or revising current data collection processes. When new data collections 
are too costly or burdensome, or if existing collections are performed so infrequently that the data are of minimal 
use, consider whether related data that are regularly and efficiently collected may be a more appropriate choice for an 
early warning system. Missing and nonstandard data, such as data from alternate assessments, may impact a system’s 
effectiveness and need to be considered when designing analytical models, performing analyses, and creating data 


rep orts. 
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Demographic Data 


The inclusion of demographic data and data on other student characteristics should be approached with caution. These 
types of data may include—but are not limited to—gender, racial/ethnic group, English learner status, special education 
program participation, and socioeconomic status. Early warning system planning teams should carefully consider 
whether these data contribute to the accuracy and predictability of a system. Planning teams must determine whether 
there is a strong correlation between these data and on-time graduation after adjusting for student risk level based on 
behavioral data, such as attendance and course performance. A student’s individual demographic data may be useful to 
consider for identifying the best intervention for that student, but care should be exercised to ensure that demographic 


data do not become a primary factor in assigning at-risk status in lieu of more predictive indicators. 


Out-of-School Data 


Early warning indicators are typically based on student attributes and behaviors that can be influenced by school-based 
interventions. If available, data on out-of-school factors can provide useful descriptive information that could help staff 
better understand a student’s individual circumstances and develop tailored interventions to accommodate these out- 
of-school factors. Out-of-school data are particularly useful when schools partner with community organizations and 
local social services agencies to provide individualized student interventions. These data include, but are not limited to, 
homelessness, involvement with the criminal justice system, involvement with social services or foster care, reports of 


substance abuse, pregnancy/teen parenthood, mobility, migrant status, and military connectedness. 


Early Warning Data Analysis 


Early warning system data are analyzed based on predetermined early warning indicators that identify the extent to 
which a student is at risk of dropping out of school. 


Analytical Models 


SEAs and LEAs use different techniques when analyzing their early warning data. Identifying and selecting which 
analytical model to use is an important step in the early warning system planning process. Rumberger and Lim’s (2008) 
analysis of 25 years of peer-reviewed studies on early warning systems found that multivariate statistical techniques, 
such as logistic regression, were commonly used to analyze early warning data. Analytical model design choices, such 
as selecting the type of model to use, which indicators to include, and which outcome to predict, are agency-specific 


decisions. 


Models may determine student risk through analysis of each indicator independently, or as part of a multivariate 
analysis. Determination of student risk based on multiple indicators can be more accurate than analyzing indicators 
independently. For example, if a single failing grade or chronic absenteeism event triggers an “at-risk” condition, many 
students who are not at high risk of dropping out may be flagged as at-risk. Multivariate analysis may be confusing for 
some users, but effective data visualization and reporting techniques can help make this approach more useful and user 


friendly. 
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Accurately Identify Student Risk 


The goal of an early warning system analytical model is to maximize the number of students correctly identified as at- 
risk while minimizing the number of students incorrectly identified, either as at-risk or not at-risk. That is, the goal is to 
maximize the number of true positive results while minimizing the number of false positives and false negatives. Early 
warning indicators that are not locally validated may lead to over- or under-identification. For example, if a districtwide 
early warning system identifies 20 percent of students as at risk of dropping out, but the district only has a 5 percent 


dropout rate, a system may be over-identifying student risk levels and creating false positive results. 


Agencies should be aware that discrepancies between an early warning system’s assignment of at-risk status and a 
district’s or state’s dropout rate may not be a result of over- or under-identification, but indicative of successful student 
intervention. For example, a district with a 10 percent dropout rate whose early warning system identifies 15 percent 
of students as at-risk may not be over-identifying students, but rather providing successful intervention programs and 


services that are resulting in reduced dropout rates. 


When establishing thresholds, it is critical to identify the level of risk at which dropping out becomes more likely than 
not for students. Lower thresholds identify more students as at risk of dropping out but may also assign some level of 
risk to students who demonstrate minimal or very low risk. Conversely, higher thresholds minimize the likelihood that 
students may be inaccurately identified as at-risk but may also miss some students who will eventually drop out. Agencies 
need to consider the tradeoffs between using broader or narrower thresholds, how many students can reasonably be 
served if flagged as at-risk, and how to prioritize students from greatest to least in need of intervention. Regardless of 
which thresholds are established, there needs to be a valid analytical approach to identify student risk and an effective 


process to refer at-risk students to interventions. 


Interpret Data 


Data interpretation can provide essential insight into why a student may be at risk of dropping out of school and should 
be included as an explicit part of the data analysis process. Interpreting early warning system data with data captured in 
a different system, such as a student information system (SIS), can point to the underlying causes of individual student 
risk, help staff better understand the specific needs of individual students, and enable the development of interventions 
that directly address students’ individual challenges and barriers. Consider a chronically absent student who is flagged 
as at-risk by an early warning system. The student may be facing a wide range of attendance barriers that are not readily 
apparent based on data captured by an early warning system. Reviewing data that are recorded in an SIS may reveal 

that the student faces transportation challenges. With this information, school staff will be better equipped to develop 
an intervention that will address the student’s specific need for reliable transportation, which will help improve the 


student’s attendance. 
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Next Generation Early Warning Systems, Indicators, Data, and Analytics 


Indicator identification, data definition, and analytical model design are not one-time activities. Indicators, elements, and 
analytical models should be iteratively refined as part of an education agency’s regular process of system improvement. 
Agencies can adapt the continuous improvement process (described in chapter 1 and illustrated in figure 1) to support 
the planning, development, implementation, and evaluation of system changes. This will help the system remain 
responsive to the evolving needs of agency stakeholders. The decision to introduce new features, indicators, data, and 
analysis techniques into a system should be determined based on whether these changes will improve the accuracy, 


effectiveness, timeliness, and usability of a system’s predictive abilities. 


Next-Generation Systems 


More robust early warning systems can identify students who experience major changes in performance but are not 

yet considered at risk of dropping out. For example, a student with a 4.0 GPA whose GPA drops to a 2.0 may not be at 
risk of dropping out but may need an intervention to help improve their academic performance. Early warning systems 
are traditionally student focused, but their data may also be used to assess organizational risk levels. Districtwide and 
statewide systems may identify schools with particularly high dropout rates, which may indicate the need for a school- 
level intervention. Similarly, if more students exhibit higher risk levels in a school that should have lower levels based on 
peer school performance, this may be interpreted as a school-level warning. The decision to use an early warning system 
in this manner should be made carefully to ensure that this approach does not negatively affect the primary purpose of a 


system: to identify students in need of intervention. 


Next-Generation Indicators 


Early warning and college readiness indicators measure related student success outcomes that can predict the likelihood 
of high school and college graduation (Allensworth, Nagaoka, and Johnson 2018). The alignment of an early warning 
system with college and career readiness (CCR) initiatives should be complementary, meaning that the system’s 

ability to identify students in need of intervention should not be compromised to support improvements to CCR 
initiatives. Incorporating CCR indicators into an early warning system could be a useful way to support student success 
post-graduation (National Forum on Education Statistics 2015). These indicators might include the presence of an 
individualized or personalized learning plan, enrollment in a career/technical education pathway program, Advanced 
Placement coursetaking, or enrollment in a dual high school/college credit course. Early warning systems can also be 
linked to a CCR information system to alert counselors when a student’s GPA falls below the threshold needed for 
entrance into the student’s college of choice. The Forum Guide to College and Career Ready Data (2015) examines how SEAs 
and LEAs are using data to support CCR initiatives. 


Next-Generation Data 


Many SEAs and LEAs have access to more sophisticated data now than in previous years, including data related to student 
satisfaction with school, student engagement, and social-emotional learning. These types of data may be useful for 
identifying students at risk of dropping out. Including pre-high school data in early warning systems, such as attendance 
data and test results from elementary and middle school, may enable earlier identification of a student who needs 
intervention services. Additionally, some early warning systems may use historical trend data to determine student risk. 
In these cases, a student who is chronically absent in only the most recent school year may be assigned a lower risk of 


dropping out than a student who is chronically absent for 3 years in a row. 
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Early warning data can also support programs and services that use similar data. Bossier Parish Schools (LA), for 
example, recently modified the early warning module in its vendor-provided SIS to help screen students for admission to 
a competitive educational program with limited enrollment. Previously, staff manually pulled attendance, behavior, and 
coursework data from the SIS, and used these data to screen and rank students using a manual rubric. Staff configured 
the early warning system module to use the early warning data to positively score and identify the strongest applicants 
for the competitive program. This process not only reduced the amount of time required to process data for the 
competitive educational program’s admissions process, it has also helped the district learn more about how the early 
warning module could be developed and implemented within the district. 


Next-Generation Analytical Models 


Next-generation analytical models are more sophisticated and may use weighted indicators to analyze data within specific 
subgroups of students; points systems, which assign points based on student performance and behaviors in relation 

to specific indicators; or tier systems, in which students are classified at low, medium, or high risk of dropping out of 
school. The classification and regression tree (CART) model is an emerging tool in early warning analytics. This method 
has shown promise identifying students at risk of poor performance on reading comprehension exams (Koon and 
Petscher 2015, 2016) and may be applicable to identifying students who are at risk of dropping out of school. 


Questions for Consideration 


It is important to determine which indicators, data, and analytics will be used in the early warning system prior to system 
development and implementation. 


What indicators will be used in the early warning system? 
Are the early warning indicators research-/evidence-based? 
Have the early warning indicators been locally validated? 
Which data will be used for each early warning indicator? 
Are these data readily available, or will your agency need to implement a new data collection? 
How frequently will the data be updated? 
How frequently and consistently will the early warning indicators, data, and analytical model be checked for 
accuracy? Will the system include automated data checks? Will the system include push notifications? 
What practices will your agency follow to ensure data quality? 
What steps will be taken to ensure system and data security? 
How will student privacy be protected? 
Which staff will be responsible for early warning data entry and analysis? 
Administrators 
Teachers 
Research staff 
Data management staff 
Student intervention and support staff 
Which analytical model will be used to identify students at risk of dropping out? 
Will the early warning system only identify students who demonstrate a risk of dropping out of school? Or will the 
system also identify and warn of major changes in individual student performance? 
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Chapter 3: 


Effective Practices in Early Warning Systems 
DY-\V/=\(o)e)aal=valen lanleltcipatciaieclalelammre| ale messy 


The effective development, implementation, and use of an early warning system require an organizational commitment 
to acting on education data through data-informed decisionmaking. 


Maximize Organizational Resources 


Multiple organizational elements—people, processes, policies, and technology—need to work together if an early 
warning system is to be developed, implemented, and used effectively. Early warning system technologies should serve 


the needs of stakeholders and support the effectiveness of organizational processes and policies. 


Education agency resources are finite. Thus, adequate time, money, and effort must be allocated for and invested in 
an early warning system if the system is to be successfully developed, implemented, and used. Constraints on agency 
resources should be considered and proactively addressed so they do not adversely impact a system’s effectiveness. 
For example, if a local education agency (LEA) does not have research, policy, or data staff who can participate in the 
early warning system planning process, the agency may be able to partner with an external organization for technical 
assistance, such as its state education agency (SEA), its Regional Educational Laboratory (REL), or a postsecondary 
institution. 


Technical Development and Implementation 


Once early warning indicators and elements have been identified, they can be used to create detailed, documented 
technical specifications and system requirements. Establishing written requirements can help assess the appropriateness 
of different system development options. Early warning systems may be designed in-house by agency staff or purchased 
from a vendor or contracting agency. LEAs in states with an SEA-operated early warning system may be able to use 

the statewide system as is or modify the system to meet the needs of local stakeholders. Aligning an early warning 
system with information systems currently in use by the agency can enable automated data imports and quality checks. 
Establishing a systematic data import, analysis, and reporting schedule supports data quality and provides staff with 
actionable data for use in the development of student interventions and supports. Prior to formal implementation, a 


system and its data should be tested for accuracy and quality. If any data errors or usability concerns arise during testing, 


they should be addressed at this time. 
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Develop and Provide Staff Training 


The introduction of an early warning system to an agency should be supported by thorough training. Staff training can 


advance three key goals: 


* Help staff understand the purpose of an early warning system 
* — Teach staff how to collect, report, analyze, and interpret data using a system 


* Prompt staff to use a system and its data in decisionmaking processes 


Training materials can be customized to meet agency-specific needs or targeted to address the information needs of 
specific departmental or staff roles. Agencies with a vendor-created early warning system may be able to obtain training 
materials from the vendor or schedule vendor-led training sessions for agency staff. Training may be offered through in- 
person training sessions, written documentation, presentation materials, online tutorials, or videos. Some staff learn best 
through written and visual materials, while others may learn best through interactive training scenarios and exercises. 


Providing multiple types of training materials can help meet the needs of staff with different learning styles. 


Training should be approached as an ongoing process to ensure that new agency staff are prepared to use the system. 
Scheduling staff training during the system development and implementation process can help identify potential barriers 
to system use. Early warning system training can also be incorporated into training materials and activities on related 
topics, such as data processes, decision support systems, or student intervention and support services. If major changes 


are made to a system while it is in use, be sure to review and update any related training materials. 


Questions for Consideration 


Early warning systems can help answer a variety of stakeholder questions. Anticipating and addressing stakeholder 
questions and information needs in training materials and resources can help encourage interest in and use of the 
system. 


Teachers: Which students are struggling in my courses and are not on track to graduate from high school? 
Counselors: Which students are at risk of dropping out or not graduating on time? 

K-12 School Administrators: How well is our school supporting all students to graduate from high school? What types 
of intervention programs should we offer to support our students? 

LEA Administrators: Which of our schools have the lowest rates of students on track for graduation? What types of 
intervention programs and resources are needed at schools in our district? 

SEA Administrators: Which LEAs have the lowest rates of students on track for graduation? 


Adapted from the Forum Guide to College and Career Ready Data (2015) 
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Report and Share Timely Data 


The frequency and purpose of data reporting and sharing activities may vary based on stakeholder information 
needs, Early warning data may be made accessible through dashboards, reports, or other outputs. For example, the 


Fairbanks North Star Borough School District (AK) regularly delivers presentations on the district’s early warning 


system to key stakeholder groups. The district has also created posters, one-page handouts, and an online dashboard 


(https: / /public. tableau.com/profile/k12northstar#!/) 
to share information and aggregate data with stakeholders. Forum Guide to Data Visualization: A Resource for 
Education Agencies 


The need for data-informed actions and interventions should be 
emphasized when reporting and sharing data. Direct data access 
for all appropriate staff can help promote data use, and actively 
disseminating data to specific individuals can help make early 
warning data actionable. For example, the Appoquinimink 
School District (DE) supports early warning data use during 
regularly scheduled meetings with senior district administrators 
and through one-on-one meetings each marking period with 
each of the district’s school principals. Supporting data drill- 


downs within a system also makes it easier for users to review 


individual student data and provide targeted interventions. 


Early warning data may also be included in regularly reviewed data reports, such as student evaluation and grade 
distribution data reports. Early warning system data can also be used to identify borderline students and/or students 
who experience major changes in behavior or performance but are not flagged as at risk of dropping out, such as a 
student whose grade point average (GPA) drops from a 4.0 to a 2.0 or a consistently attending student who becomes 
chronically absent. More robust data systems can generate additional reports that measure these types of major changes 


in student performance, which could then be used to identify students in need of early interventions. 


Use Data to Inform Student Interventions 


Early warning systems are more than just a tool to identify Forum Guide to Taking Action with Education Data 
students who may eventually drop out of school. Early 

warning systems facilitate the use of education data in a manner 
that improves an education agency’s ability to 

provide interventions, services, and supports to help students 
whose educational outcomes are at risk. It is important 

to ensure that a system and its data are used appropriately 

and connected to actions that improve student outcomes. 
Evaluating current mechanisms by which at-risk students are 
identified for interventions can identify opportunities for an 
early warning system to support current practices, programs, 


and processes. 
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Stakeholder groups may need different types of data to inform action. As illustrated in figure 2, early warning system 
data may be used differently across an education agency based on the specific information and data needs of stakeholder 
groups. Staff should be encouraged to engage in data-informed discussions and act using early warning system data. 
Leveraging existing team-based structures and collaborative activities, such as team meetings, leadership meetings, and 


professional learning communities, can promote early warning system data use. 


SEA LEADERSHIP CENTRAL OFFICE SCHOOL ADMINISTRATION 


* Monitor statewide ADMINISTRATION 


dropout rate 

« Inform planning and 
policy decisions 

¢ Provide training and 


* Monitor district dropout rate ¢ Monitor school dropout rate 
* Support districtwide data use * Support schoolwide data use 


+ Implement districtwide pinilenre res neowde 
intervention programs 


support services to LEAs intervention programs 


DATA MANAGEMENT FINANCE AND BUDGET RESEARCH AND 
ASSESSMENT 


* Compile, analyze, and ¢ Calculate return on investment ¢ Monitor trends in student risk 


report data * Allocate funds to program * Evaluate system predictiveness 
¢ Assess data quality 


STUDENT SERVICES CURRICULUM AND TEACHERS 
INSTRUCTION 


¢ Identify students in need * Identify students in need of * Identify at-risk students 
of behavior intervention academic intervention ¢ Refer students to targeted 

* Provide targeted * Provide academic support intervention services 
intervention services services 


SCHOOL BOARD COMMUNITY PARTNERS 


* Refine policies * Support in-school 

* Inform budget interventions with 
allocations and funding out-of-school programs 
decisions and services 


Figure 2. Representative examples of how early warning system data are used by stakeholders 
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The National Survey on High School Strategies Designed to Help At-Risk Students Graduate found that approximately 
half of the high schools that used an early warning system reported limited coordination between the system and the 
provision of other school services, including intervention programs (U.S. Department of Education 2016). Developing 
a culture of data use, encouraging data-informed decisionmaking, and clearly linking specific intervention programs and 


services to student risk factors can help improve coordination between early warning systems and other school services. 


Resource Spotlight: Dropout Prevention and Intervention 


Several agencies and organizations provide resources and assistance on dropout prevention and intervention. SEAs 
and LEAs may find it useful to consult these resources as they consider how to best provide intervention programs and 
services to students who are at risk of dropping out of school. 


Diplomas Now 

http://diplomasnow.org/ 

Diplomas Now, funded in part by a U.S. Department of Education Investing in Innovation Fund (i3) grant, is a program that 
helps middle schools and high schools develop a comprehensive plan to improve student achievement. Diplomas Now 
partners with schools to set goals, develop a strategic plan, implement an early warning system, and engage in regular 


data reviews. The program also provides curriculum, teacher coaching, and student support to help each student get 
back on track. 


Everyone Graduates Center 

http://new.every1graduates.org/ 

The Everyone Graduates Center, a unit of the Johns Hopkins University School of Education, seeks to identify the barriers 
that stand in the way of all students graduating from high school prepared for adult success, develop strategic solutions 
to overcome these barriers, and build local capacity to implement and sustain these strategic solutions. The center 
offers multiple resources on the effective development, implementation, and use of early warning systems, including 
compilations of resources used by state partners. 


What Works Clearinghouse 

https://ies.ed.gov/ncee/wwc 

Resources on research-based interventions are available through the What Works Clearinghouse (WWC). WWC reviews 
existing research on different programs, products, practices, and policies in education to provide educators with the 
information they need to make evidence-based decisions. The WWC guide Preventing Dropout in Secondary Schools 
(Rumberger et al. 2017) provides recommendations that focus on reducing high school dropout rates. 
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Letters from the Field - Using an Early Warning System to Support Student Interventions 


The Fairbanks North Star Borough School District (AK) recently hired four new social service managers, who started a few 
weeks into the Spring semester 2018. The managers’ primary role is to coordinate, implement, and provide interventions 
and services to students whom the district’s early warning system identifies as at high or medium risk of dropping out. 
The catalyst for creating these new staff positions came from a presentation to the Fairbanks North Star Borough School 
Board. This presentation focused on the work and evidence of success with a former grant program using the district’s 
early warning system and showed that focused interventions could reduce the district’s dropout rate and reduce dropout 
rate gaps between demographic groups. 


The social service managers have already benefitted from using the district’s early warning system to inform their work. 
The managers noted that the early warning system was especially useful in establishing their caseloads by meeting 
with counselors and principals to review children at high risk of dropout. The managers have worked with many children 
who would have fallen through the cracks if not for the early warning system. Their department found that students who 
were considered at risk of dropping out were not consistently prioritized for intervention services. The early warning 
system reminds school staff that the most vulnerable children are the population who will make the greatest gains with 
intervention. 


The social service managers have implemented numerous successful interventions during their first semester, including 
the following: 


e Astudent with runaway status and poor mental health received counseling, a recommendation to a community 
social service agency, and assistance to help meet basic food and clothing needs. The student stabilized ata 
community agency and was able to receive residential behavioral health services and to improve attendance. The 
family increased positive engagement with the school, and the student received personalized supports for social and 
emotional needs. 

Information from the early warning system helped identify a student who was at risk of dropping out. The student 
disclosed information—including that the student lived in a car and had limited access to food and basic needs— 
that qualified the student and the student’s siblings as eligible for McKinney-Vento services.* Collaboration 

between social service managers helped the family receive assistance with transportation, basic needs, and 
recommendations to community supports. 

A high school student with multiple risk factors faced several barriers, including homelessness, financial stress, a 
lack of transportation, and minimal parental support. The student expressed a goal to graduate from high school but 
recognized the unlikelihood of overcoming current barriers without support. In addition, because of the student’s age 
and independence, the student felt isolated in a traditional high school setting. Upon exploration, the student agreed 
to enroll at the Career Education Center, and the student’s academic performance and truancy patterns improved. 
The Career Education Center is a self-paced program for students who have not been successful in the district’s 
traditional high schools. Students in the program may earn a diploma by meeting the same requirements that all 
students in the district must meet, independently and at their own pace. 


* For more information, see the McKinney-Vento Education for Homeless Children and Youths Program. 
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Evaluate System Effectiveness 


System evaluation provides critical information on system strengths and weaknesses, which can be used to implement 


changes that support continuous system improvements. Regular evaluation of an early warning system may focus on the 


following areas: 


System accuracy — The analytical model and early warning indicators need to meet agency standards 
regarding the accurate identification of students in need of intervention. Early warning system data that 

are ready for retirement may be used to evaluate how effectively the system identifies students in need of 
assistance. Integrating systems can also be a useful strategy; for example, if a student is assigned a dropout 
code in the student information system, an automated system check could determine whether that student 
was previously flagged in an early warning system. 

System performance — Routine system maintenance, such as quality assurance checks and security audits, 
can identify needed system improvements. 

System utility and actionability — A system’s usefulness in helping staff and stakeholders identify students 
in need of intervention can be evaluated in multiple ways. User statistics, such as login data, can be used to 
determine how frequently and consistently a system is used by different stakeholder groups. Surveying users 
can solicit feedback on the accessibility and usefulness of an early warning system. Engaging an external 
agency in the evaluation process may solicit more candid responses from users on a system’s benefits and 
challenges. 

Intervention effectiveness — Evaluating intervention outcomes will determine whether interventions 

are successfully helping students who were at risk of dropping out. Monitoring students who are no longer at 
risk of dropping out can also be useful in determining the effectiveness of specific interventions. SEAs are less 
likely to be involved in evaluating student-level interventions than LEAs, as they do not work directly with 
students. 


Once needed changes and improvements to a system are identified, a plan should be developed to support the 


implementation of changes and improvements. Agencies can adapt the continuous improvement process (described in 


chapter 1 and illustrated in figure 1) to support the planning, development, implementation, and evaluation of system 


changes. If a system was built in-house by the agency, agency staff can more readily act to improve the system based on 


evaluation data. Agencies with an externally developed system can coordinate with the system’s creators—vendors, 


organizational partners, or others—to implement system changes based on evaluation data. 
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Questions for Consideration 


There are many factors that contribute to the success of an early warning system. Developing a plan for how your agency 
will Support the early warning system development, implementation, and use process can help ensure that the system 
will be beneficial to users and inform student interventions. 


How will training be provided to agency staff? Will training be provided before, during, or after implementation? 

How will the early warning data be disseminated to users? Will data be available to users through a report, 
dashboard, or some other method? 

How will student privacy be protected when reporting, sharing, and using data? 

Who will be the primary users of the early warning system, and for what purposes will they use early warning data? 
How will the early warning data be communicated to stakeholders? 

2 Internally to agency staff and administrators 

2 Externally to other education agencies 

5s Publicly to parents, community partners, and other local and regional stakeholders 

Does your agency have a communications plan? 

Which staff will be responsible for reporting early warning data? 

How will your agency encourage the systematic use of early warning data? 

Are there formalized structures and processes in place to support system implementation, sustainability, and use by 
a variety of stakeholders? 

Are there specific processes in place for when the early warning system identifies students in need of interventions? 
Which staff will be involved in these processes? 

Do specific intervention programs and services exist to help students identified through the early warning system? 
How will the effectiveness of the early warning system be evaluated? 
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Chapter 4: 


Case Studies From State and Local 
mo [Ulere ldo) amatex-1alelters) 


This chapter provides an overview of how early warning systems are used in different state and local education agencies 
(SEAs and LEAs). The seven case studies included in this chapter offer a closer look at how SEAs and LEAs have 


implemented, or are in the process of implementing, an early warning system. 


State and Local Early Warning Systems 


The planning, development, implementation, and use of early warning systems is enhanced when SEAs and LEAs work 
together. LEAs can provide their SEA with useful insights into local needs, issues, and concerns, which the SEA can take 


into consideration when planning a statewide early warning system. 


SEAs can provide LEAs with support in a variety of ways. SEA research staff may be able to assist LEAs with indicator 

identification and validation if the LEA does not have a researcher on staff. Statewide early warning systems developed by 
the SEA can provide LEAs with a set of indicators and data that can be used to create a local-level system that meets local 
stakeholder needs. In schools and districts that already have a local early warning system, it could be useful to review the 


statewide early warning system to see which aspects of the statewide system overlap and support the local system. 


While both statewide and local early warning systems are designed to support improved student outcomes, SEAs and 
LEAs approach these systems in different ways. Statewide early warning systems can be used to inform state education 
policy, monitor statewide dropout risk rates, and help LEAs improve graduation rates. Statewide early warning systems 
include data on more students than local early warning systems, which can reveal important trends that wouldn’t be 
visible at the LEA level. This can also provide LEAs with information on students who transfer between districts within 
the state, which can help ensure that transfer students in need of support receive intervention in a timely manner. SEAs 
may develop guidelines, processes, or requirements that govern how LEAs report data for use in a statewide early 
warning system, but requiring all LEAs in the state to use the statewide system is uncommon. Additionally, some SEAs 


can develop a statewide early warning system without being able to directly implement student-level interventions. 


Additional information on statewide early warning systems is available from the Statewide Longitudinal Data System 
Grant Program’s Development and Use of Early Warning Systems (Curtin, Hurwitch, and Olson 2012). 


Case Studies 


This chapter presents seven case studies that detail the actual experiences of SEAs and LEAs that have implemented, 
or are in the process of implementing, an early warning system. The case studies are presented in alphabetical order by 
state; for states with both an SEA and LEA case study, the SEA case study is presented first. A summary of the lessons 
learned from these case studies is included at the end of this chapter. 
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In broad terms, most of the case studies address the following aspects of the early warning system planning, 
development, implementation, and use process: 


* Overarching descriptive information about the agency 

* Description of the system planning, development, and implementation process 

* Review of system indicators, data elements, analytical approach, and data quality practices 

* Discussion of system and data use, including the development and implementation of student interventions 
¢ Lessons learned 


These case studies are presented as examples that schools, districts, and states can learn from as they begin planning for a 
new, or continue to use an existing, early warning system. SEAs and LEAs who are planning to implement, or are in the 
process of implementing, an early warning system may find it useful to learn about how another agency developed and 
used an early warning system. 
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Case Study: Fairbanks North Star Borough School District, AK 


The Fairbanks North Star Borough School District created an early 
warning system, called the Dropout Risk Model, in the 2009-10 school 
year to improve the effectiveness of its Graduation Success Program. The 
Graduation Success Program provided targeted interventions to students 
in kindergarten through 12th grade and received funding through the 
American Recovery and Reinvestment Act of 2009 in the program’s last 
2 years of operation. While the Graduation Success Program concluded 
at the end of the 2010-11 school year, the district continues to use the 
Dropout Risk Model as a tool to help ensure that students who are at 
high risk of dropping out of school receive the interventions they need to 


succeed. 


System Planning and Implementation 


In the 2006-07 school year, the Fairbanks North Star Borough School 
District implemented the Dropout Prevention Program (later renamed 
the Graduation Success Program) to address the needs of students who 
were at risk of dropping out of school. In the 2008-09 school year, 

a district researcher participated in the program’s annual evaluation 
process. The researcher’s analysis of dropout rate data showed that a 


statistically comparable group of students not served by the program 


Agency Background 


Fairbanks North Star Borough School 
District serves approximately 14,000 
students enrolled in 35 public schools. 
The adjusted cohort graduation rate is 
74 percent. Approximately 60 percent of 
the student population is categorized in 
the White racial/ethnic group, while the 
next largest student groups are the Two 
or More Races group (14 percent) and 
the American Indian/Alaska Native group 
(9 percent). Approximately 35 percent 
of students qualify for free or reduced 
lunch, 16 percent have an Individualized 
Education Program (IEP), and 3 percent 
are English learners. (Source: ED, NCES, 
CCD, School Year 2015-16; EDFacts, 
School Year 2015-16). Note: These 
numbers represent the most recent ED 
data and do not necessarily reflect the 
latest district data. 


were twice as likely to drop out of school when compared with students who were served by the program. These 


findings also identified an opportunity to dramatically improve the program’s effectiveness in reducing dropout rates 


using an early warning system. The analysis showed that the district dropout rate was predicted to have been 6.0 percent 


had the program not been in place; 5.4 percent with the program in place; and 4.2 percent if the program had used an 


early warning system to make service decisions. 


Based on these findings, the researcher recommended continuing the Graduation Success Program and creating a 


new early warning system, the Dropout Risk Model, to inform the provision of program services to students. It was 


predicted that changing the population of students served by the program would more than double the program’s 


effectiveness in reducing district dropout rates. Prior to the implementation of the Dropout Risk Model, students 


were identified as in need of program services based on vague criteria and unsystematic referrals. District leadership 


decided to implement the changes recommended by the program evaluation. School and district staff and administrators, 


including district superintendents, assistant superintendents, school principals, data management staff, research staff, 


and Graduation Success Program staff participated in planning for and implementing the changes recommended in the 


program evaluation. The Dropout Risk Model was developed in-house by the district researcher, who had knowledge and 


expertise in statistical modeling. The researcher was solely responsible for creating the Dropout Risk Model, including 


determining which indicators would be used. 
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Early Warning Indicators, Data, and Analytics 


The Dropout Risk Model uses eight early warning indicators to calculate each student’s dropout probability. These 
indicators focus on attendance, course and assessment performance, on-track progression, and out-of-school 


suspensions. 


Indicator definition and determination was guided through local Early Warning Indicators 


validation using internal agency data. Additional indicators were 
also suggested by staff and the published research on early warning Attendance rate 

Difference between high school credits 
and number of years in high school 
District assessment performance 


indicators. A statistical analysis of 10 years of longitudinal data was 


performed to identify indicators for inclusion in the model. Several 


indicators were considered in this analysis, but only indicators that Former dropout 
reduced the false positive rate and increased the true positive rate Grade point average (GPA) 
were included in the model. For example, students with high rates of Retention 


State test performance 
Out-of-school suspensions 


tardiness showed higher dropout rates in general, but since tardiness 
data did not reduce the false positive rate and increase the true positive 
rate, these data are not included in the model. Early Warning Indicator Data Elements 


Aggregate daily attendance 
Aggregate daily membership 

student demographic data. Ultimately, any indicators related to student Accrued high school credits 
demographics or factors outside of the control of the student or the Number of years in high school 
National percentile rank for AIMSWeb 
reading assessment 

National percentile rank for AIMSWeb 
math assessment 

determines student risk based on what students do, not who they are. Former dropout 


Grades issued in each course 


Another important model design consideration was whether to include 


school were excluded from the model, including data on gender, race/ 
ethnicity, economic disadvantage, homelessness, migrant status, English 


learner, military connectedness, and disability status. Thus, the model 


No new data collections were necessary to support the Dropout Retained designation 
Proficiency on state standards-based 


math assessment 


Proficiency on state standards-based 
Any data not included in this report are extracted from the student reading assessment 


Risk Model, as all the data elements were already available. Most 
of the data are included in the district’s annual report to the SEA. 


information system (SIS) or sourced from preexisting data files, such Proficiency on state standards-based 
as state assessment reports. Additionally, certain data are compiled and science assessment 

Proficiency on state standards-based 
writing assessment 

Out-of-school suspensions 


transformed for inclusion in the model. For example, kindergarten 


through third-grade students do not receive letter grades using the A, B, 


C, D, F grading system, but grading data for these students are aligned 
with a 4.0 grading scale for inclusion in the Dropout Risk Model. 


District technology staff support the district’s data processes and are involved in collecting, disseminating, and displaying 
the data. Additional departments and staff participate in the data collection process as appropriate; for example, student 
grades are submitted through the district’s SIS and daily attendance is tracked by attendance secretaries in each school. 
Each data file used in the Dropout Risk Model is heavily vetted for accuracy and several departments participate in 


reviewing the district’s annual report prior to submission to the state. 


Chapter 4: Case Studies From State and Local Education Agencies 27 


The analytical model was developed over a few months and implemented during the 2009-10 school year. The model 
assigns students a risk level of low, medium, or high using a custom analytical approach. First, an initial risk level is 
calculated through a logistic regression model of two indicators: attendance and grade point average (GPA). Then, the 
remaining early warning indicators, each of which have established independent cut-points, are incorporated into risk 
level determination using a process similar to classification trees. This custom analytical model retains the complexity 
necessary for sufficient reliability and validity but is easier for users and stakeholders to understand than a multivariate 
logistic regression model. The model is designed to increase the true positive rate and decrease the false positive rate 
through reliable and valid analysis, and identify a smaller number of students through increased, but understandable and 
transparent, model complexity. The district researcher who created the model performs all data analyses and reporting. 
Since initial implementation, minor changes have been made to the model based on staff feedback and impact analysis 


results. 


Risk level data for each student in grades K-12 are stored in a data file, which technology staff upload into the SIS. The 
SIS’s primary screen displays each student’s risk level using a custom color-coded dot icon (a green dot represents low 
risk, yellow represents medium risk, and red represents high risk). A data file, which includes risk-level data, is also sent 


to each principal each year. 


System Use 


After the Dropout Risk Model was implemented in the 2009-10 school year, the district needed to determine how the 
Graduation Success Program would use the model to determine which students would receive services. At the time, 19 
program staff were responsible for serving 1,000 students. The ratio of staff caseload capacity to students who would 
potentially benefit from program services was quite high, which limited the program’s ability to serve all students 
identified as medium and high risk. The district’s data showed high-risk students comprised only 12 percent of student 
enrollment but comprised roughly two-thirds of all district dropouts. Furthermore, a propensity score matching of 
student data showed a comparable 50 percent reduction in dropout rates across all 3 risk categories, meaning that 
students at high risk of dropping out were equally likely to benefit from program services as those students who 

were at medium and low risk. To maximize the program’s effectiveness in reducing the district dropout rate, district 
administration set the expectation that the program was to exclusively serve students who were identified by the model 
as high risk. An implementation review was also conducted to verify the program’s conformity with this change. This 
expectation helped ensure that the program’s limited resources were allocated to the group of students who would have 


the strongest influence on district dropout rates. 


School principals, social service managers, counselors, and prevention/intervention specialists use the model’s risk-level 
data to prioritize and assign students to interventions, services, and programs. To further encourage support for and 

use of the model, the model creator has delivered presentations to key leaders and stakeholders, including principals, 
federal program staff, Graduation Success Program staff, community achievement focus groups, and the School Board. 
These presentations have been particularly effective at generating support for the model. As an example, a presentation 
to the School Board showed that focused interventions could reduce the district’s dropout rate and reduce dropout rate 
gaps between demographic groups. Furthermore, each student that remained in school ensured continued base student 
allocation funding for that student, making the model a positive return on investment for the district. This presentation 


served as a catalyst for the Board to consider specific methods to implement targeted interventions and motivated the 
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district to hire four new social service managers. The primary role of these new staff is to coordinate, implement, and 


provide interventions and services to students who are identified as high and/or medium risk. 


Since the model was first implemented, its effectiveness has been evaluated based on the district dropout rate. 

The served student group dropout rate is compared with the dropout rate of a comparison group defined through 
propensity score matching. Sufficient evidence has shown that the Graduation Success Program’s services to high-risk 
students directly reduced the districtwide dropout rate, and the district dropout rate has remained consistently lower 
for students who received services than a comparable group of students who did not receive services. In the first year 
that data generated by the Dropout Risk Model were used to assign students to services and interventions provided by 
the Graduation Success Program, the district dropout rate was 4.2 percent, the lowest recorded dropout rate for the 


district. In the following school year (2010-11) the dropout rate was even lower, at 3.9 percent. 


Lessons Learned 


* Balance complexity and transparency. The creator of the Dropout Risk Model noted that model 
design choices are especially important when creating an early warning system. They found that the use of 
certain statistical methods can make the analytical model more complex without drastically compromising 
transparency, which can help overcome some of the barriers associated with using a complex analytical model. 
* Devote adequate time to model development. The district’s development and implementation 
of the early warning system was relatively quick, allowing only a short period of time to develop the 
analytical model. As a result, multiple model refinements were necessary after the Dropout Risk Model was 
implemented. The model creator recommended that early warning system planning teams budget a sufficient 


amount of time to the development of the early warning system analytical model. 
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Case Study: Delaware Department of Education 


The Delaware Department of Education (DDOE) recently created an 
Early Warning System (EWS) plug-in as part of the state’s Education 
Insight (EdInsight) Program. EdInsight, which was initially funded under 
Race to the Top in 2010, is a collection of projects and systems that 
support data-driven decisionmaking throughout the Delaware education 
system. The EdInsight Program has been in place for several years and 
includes Delaware’s statewide longitudinal data warehouse and statewide 
performance management dashboard. The EWS plug-in is one of several 
projects that seek to leverage the EdInsight Dashboard to improve 


outcomes for all Delaware students. 


System Planning and Implementation 


The EWS planning and development process began in September 2017. 
The process was a collaborative effort involving SEA and LEA staff and 
external organizations. DDOE information technology (IT) and program 
staff worked with the Ed-Fi Alliance throughout the planning and 
implementation process. The Ed-Fi Alliance, funded by the Michael and 
Susan Dell Foundation, is a community of educators, technologists, and 


leaders committed to ensuring that schools, districts, and states can see, 


Agency Background 


The state of Delaware serves 
approximately 135,000 students enrolled 
in 227 public schools within 53 school 
districts. The adjusted cohort graduation 
rate is 86 percent. Forty-six percent of the 
student population is categorized in the 
White racial/ethnic group, while the next 
largest student groups are the Black or 
African American group (31 percent) and 
the Hispanic/Latino group (16 percent). 
Approximately 37 percent of students 
qualify for free or reduced lunch, 15 
percent have an IEP, and 7 percent are 
English learners. (Source: Source: ED, 
NCES, CCD, School Year 2015-16; ED, 
OESE, School Year 2014-15). Note: These 
numbers represent the most recent ED 
data and do not necessarily reflect the 
latest district data. 


secure, and use education data. The EWS plug-in was originally developed by the Pennsylvania Department of Education 


(PDE) and was made available to Delaware through the Ed-Fi Exchange. Delaware worked with an Ed-Fi consulting 


group to adopt and modify the plug-in to meet Delaware’s needs. Use of the plug-in incurred no major cost to the 


DDOE, although there was a small cost to incorporate the plug-in into the EdInsight Dashboard. 


DDOE worked with an oversight committee to help guide goal setting, project planning, communications, and product 


review processes. Individual SEA and LEA staff participated in the EWS development process through a formal 


stakeholder engagement program. DDOE conducted two focus groups with invited LEA representatives. These focus 


groups included discussion on the topic of dropout prevention, demonstration of the EWS plug-in developed by PDE, 


and discussion on the relevance of the system to Delaware. A dedicated EdInsight data coach also collected formative 


feedback on the plug-in via interviews to help identify potential enhancements to the plug-in. 


System integration and testing occurred in Winter 2018 prior to statewide release. DDOE worked with an external 


vendor to incorporate the EWS plug-in into the EdInsight Dashboard. A DDOE IT developer is responsible for vendor 


coordination, and a DDOE analyst is responsible for ensuring data quality and accuracy. DDOE’s established and stable 


software application deployment process and relationship with the Ed-Fi Alliance has resulted in faster development time 


and reduced cost to the agency. 


Early Warning Indicators, Data, and Analytics 


The indicators used in the EWS plug-in were selected based on research conducted for PDE by a leading early warning 


researcher. This research identified the “ABC” indicators (attendance, behavior, and course performance) as critical in the 
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early identification of students who are at risk of dropping out of school. DDOE reviewed these indicators with district 


and school staff during the EWS planning process to ensure that LEAs understood how the indicators are applicable in 


Delaware. 
Early Warning Indicators 
The EWS requires no new data collections or data entry activities to Attendance 
support implementation and use. Instead, the EWS plug-in uses data Behavior 
that are stored in the statewide SIS. Each day, the prior day’s data are Course Grades in Language Arts 
extracted from the SIS, loaded into the state data warehouse, and then Course Grades in Mathematics 


transformed and loaded into the EdInsight Dashboard. Several data 
standardization checks are performed throughout the extract, transform, and load process. DDOE also performed a 


series of thorough data checks in alignment with the agency’s agile system development processes. 


The EWS plug-in automatically flags students as at risk of dropping out of school if student data related to at least one 
at-risk indicator are found to be above a predefined threshold. No additional data analytics are required for the plug-in 
to identify whether a student is at-risk. The EdInsight Dashboard uses course grade metrics to measure major changes in 


student performance, but these metrics are distinct from the course grade indicators used in the EWS. 


In February 2018, the EWS plug-in was deployed statewide and made available to approximately 13,000 active EdInsight 
Dashboard users. The EWS will help LEAs quickly identify whether a student has been flagged as at-risk, including 
students who recently moved between districts and schools within the state, which will enable LEAs to provide 


interventions to students. 


System Use 


The DDOE has several mechanisms in place to support LEA use of the EWS plug-in. Documentation on how to use 
the EdInsight Dashboard and EWS plug-in is easily accessible online, and the DDOE has created a plan to guide the 
agency’s EWS-related communication activities. DDOE has a dedicated EdInsight data coach who provides free on-site 
staff training to every LEA during professional learning community time and staff meetings; these training sessions will 


include information on how to use the EWS plug-in. 


Presentations on the EWS plug-in will also be delivered at the DDOE’s regularly scheduled monthly cadre meeting. 
LEA staff, including data managers, curriculum directors, school counselors, IT directors, and lead teachers, attend the 
monthly cadre meetings. The DDOE regularly encourages EdInsight Dashboard use during cadre meetings and often 
incorporates a presentation on the Dashboard into the meeting. These meetings provide LEAs with the opportunity to 


share how they use data to drive local decisionmaking processes. 


The DDOE anticipates LEA feedback on the EWS during future monthly cadre meetings and EdInsight training sessions. 
DDOE expects that this feedback will be used to inform continued developments and improvements to the EdInsight 
Dashboard and EWS plug-in. The DDOE hopes that LEAs will be motivated by the EWS plug-in to develop a portfolio 
of resources and interventions to assist students and reduce their risk of dropping out of school. In support of this goal, 
DDOE is considering working with the University of Delaware to provide resource recommendations and opportunities 
to LEAs. 
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Lessons Learned 


* Collaborate with trusted partners. The time necessary for system development was greatly reduced 
due to the DDOE’s collaborative partnership with the Ed-Fi Alliance. The help provided by the Ed-Fi Alliance 
enabled the DDOE to implement the EWS plug-in in a short timeframe and at a significantly reduced cost. 

* Solicit feedback to ease system implementation. DDOE has taken steps to support the ease and 
effectiveness of the system implementation process. The Department pilot tested the system with select LEAs 
throughout the state, and feedback from participants was used to make system changes and improvements 
prior to statewide release. 

* Communicate early and often. DDOE found it useful to share information about the system with LEAs 
during the state’s monthly cadre meetings. This provided LEAs with the opportunity to learn about the 
benefits of the system, which built anticipation leading up to the EWS’s statewide release. 
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Case Study: Appoquinimink School District, DE 


The Appoquinimink School District is very focused on education and 

has strong support from community stakeholders, which has helped the 
district maintain a consistently high graduation rate. Although the district 
dropout rate is consistently low, district staff and administrators were 
interested in better understanding what factors may influence a student’s 


decision to drop out of high school. 


System Planning and Implementation 


The Delaware Department of Education (DDOE) recently developed 

a statewide early warning system for LEAs to use. The district used 

this work by the DDOE as an opportunity to explore which early 
warning indicators are predictive of student dropout within the district. 
The statewide DDOE-created early warning system uses attendance, 
behavior, and course grades in language arts and mathematics to predict 
student risk of dropout. The district has used these DDOE-developed 
indicators as a baseline for developing a Student Index Warning, which 
serves as the district’s first step in locally validating potential early 
warning indicators. The district is planning to locally validate these 


indicators, but has not yet done so. The Student Index Warning will be 


Agency Background 


Appoquinimink School District serves 
approximately 10,000 students enrolled 
in 16 public schools. The adjusted 
cohort graduation rate is 90 percent. 
Approximately 60 percent of the student 
population is categorized in the White 
racial/ethnic group, while the next 
largest student groups are the Black or 
African American group (25 percent) and 
the Hispanic/Latino group (7 percent). 
Approximately 15 percent of students 
qualify for free or reduced lunch, 14 
percent have an IEP, and 2 percent are 
English learners. (Source: ED, NCES, CCD, 
School Year 2015-16; EDFacts, School 
Year 2015-16). Note: These numbers 
represent the most recent ED data and 
do not necessarily reflect the latest 
district data. 


used to compare data from students who drop out of high school and students who graduate from high school. This 


comparative analysis will help determine what differences exist between these two groups of students and which early 


warning indicators are predictive of student dropout within the district. 


Early Warning Indicators, Data, and Analytics 


The Student Index Warning uses attendance, discipline, academic performance, and retention indicators to calculate a 


dropout probability score for each student. 


In addition to data used in DDOE’s early warning system, such as chronic absences and course failures, the Student 
Index Warning also draws on data available within the district. Both Smarter Balanced and MAP® Growth™ assessment 


data, as well as retention data, are included in the Student Index Warning. Data on student demographics, military- 


connected status, choice status, and transient status are also included. The Student Index Warning also tracks each 


student’s grade of entry into the Appoquinimink School District, with a focus on whether students attended pre- 


kindergarten or kindergarten in the district. These data will be used to examine whether students who begin their 


educational career in the district are more likely to graduate than students who move into the district at a later grade 


level, with the goal of understanding what factors within district schools contribute to student success. 


The Student Index Warning uses student data to calculate a points-based dropout probability score for each student. 


Each data element has an associated point system, and points from each data element are weighted equally in calculating 


dropout probability scores. For example, student grades are calculated using a 4-point grade point average scale: A 


grades are 4 points, B grades are 3 points, C grades are 2 points, and F grades are 1 point. Attendance is weighted using 
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an equivalent 4-point scale: Students with an attendance rate of 95-100 
percent receive 4 points. By monitoring a wide variety of data over 
time, with each data element using a 4-point scale, the district will 

be able to determine which factors are valid predictive indicators of 


dropout for students enrolled in the district. 


The Student Index Warning exists as part of the district’s data files. 
Each school within the Appoquinimink School District has its own 
spreadsheet file of school-level data. The file includes 17 worksheets, 
one of which is the Student Index Warning worksheet. Each worksheet 
includes key data and statistics on a specific topic of interest, such as 
grade distributions, teacher evaluations, and final grade correlation 
data. Pivot tables in each worksheet enable connections between the 
file and other data sources. For example, when a user clicks on a data 
cell that shows a percentage figure of chronically absent students, the 
user can view a list of all chronically absent students and review each 


student’s individual records. 


All Delaware districts and schools (public, charter, and alternative) 
use the same SIS, which helps facilitate data sharing between individual 
schools, their district central office, and DDOE. The Appoquinimink 
School District’s central office Assessment, Accountability, and 


Educational Data Department staff manage the district’s data systems 


Early Warning Indicators 


Attendance 

Discipline 

Academic performance 
Retention 


Early Warning Indicator Data Elements 


Chronic absences 

Number of discipline referrals 
Number of suspensions (in-school and 
out-of-school) 


Course grades and failures 

Rasch Unit (RIT) scores from MAP® 
Growth™ tests 

Smarter Balanced assessment scores 
Retention 


Other Data Elements 


Demographics 

Choice status 

Military connectedness 
Transient status 

Grade of entry 


and assist with exporting data from the SIS and importing the data into the district’s data files. The data files, including 


the Student Index Warning, are updated each marking period; middle schools and high schools operate on a quarter 


marking period system, while elementary schools use a trimester marking period system. The district has iteratively 


developed and refined the file format and data update process over the past few years. 


System Use 


The Assessment, Accountability, and Educational Data Department supports school use of the Student Index Warning 


through regularly scheduled “data dig” meetings. The assessment, accountability, and educational data coordinator 


also reviews these data in meetings with senior district administrators, including the superintendent, assistant 


superintendents, director of instruction, and special education supervisor. 


The assessment, accountability, and educational data coordinator travels to each school in the district to review the 


school’s data together with the principal. Data dig meetings are held each marking period and are approximately 


two hours long. By providing a wide variety of data in one easy-to-view file, principals can view their school’s data in 


aggregate, identify their school’s strengths and challenges, and act to address issues in need of further attention. 


After data dig meetings, school principals meet with school psychologists and response to intervention (RTI) teams to 


advise them of students who were identified by the Student Index Warning as in need of intervention. Interventions are 


provided by school RTI teams and student services staff. 
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Lessons Learned 


¢ Draw on community connections. The assessment, accountability, and educational data department 
coordinator noted that connections within the education data community have been very helpful in developing 
and refining the Student Index Warning. Connecting with colleagues in the education data community has 
helped the Assessment, Accountability, and Educational Data Department learn about the barriers, strategies, 
and successes other states and districts have faced in developing, implementing, and using an early warning 
system. 

* Diversify data teams. The Assessment, Accountability, and Educational Data Department staff come from 
diverse professional backgrounds. Some staff have professional experience in data analysis, accountability, 
assessment, and research, while other data staff have professional experience as educators and school 
administrators. This diversity of staff background has helped the Department better understand how they can 
help support school administrators and staff fulfill their roles and responsibilities, which has enhanced the 


Department’s ability to provide data that are easy to use in analysis, evaluation, and decisionmaking. 
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Case Study: Montana Office of Public Instruction 


The Montana state education agency, the Office of Public Instruction 
(OPI), was interested in lowering the state dropout rates for all students, 
especially among certain subgroups. To help schools lower their dropout 
rates and encourage data use among Montana schools, the OPI created a 
voluntary statewide early warning system, Montana EWS, in the 2011- 
12 school year. The Montana EWS was created in-house by an OPI 


research data analyst. 


System Planning and Implementation 


Montana schools are not required to use the Montana EWS; instead, 
each school decides whether to implement it. An organizational culture 
that recognizes the importance of the Montana EWS and prioritizes its 
use is an important factor in whether a school decides to implement the 
Montana EWS. 


It is sometimes challenging to convince schools that the Montana EWS 
is an important tool since school use of the Montana EWS is voluntary, 


is not required by any OPI programs, and is not attached to any funding 


Agency Background 


The state of Montana serves 
approximately 145,000 students enrolled 
in 830 public schools in 497 districts. 
The adjusted cohort graduation rate is 
86 percent. Approximately 79 percent 

of the student population is categorized 
in the White racial/ethnic group, while 
the next largest student groups are the 
American Indian/Alaska Native group (11 
percent) and the Hispanic/Latino group 
(4 percent). Approximately 45 percent of 
students qualify for free or reduced lunch 
and 12 percent have an IEP. (Source: 
Source: ED, NCES, CCD, School Year 
2015-16; ED, OESE, School Year 2014- 
15). Note: These numbers represent 

the most recent ED data and do not 
necessarily reflect the latest district data. 


sources. The OPI has used several strategies to encourage schools to implement and use the Montana EWS. The OPI 


has worked closely with several schools to help connect the Montana EWS to interventions that are currently in place 


in the school. Additionally, the OPI offers small grants to encourage EWS use and help schools implement student 


interventions. The Montana EWS was also a part of Graduation Matters Montana, a program that provided private grant 


money to communities for projects that encourage students to graduate from high school. Looking ahead, the Montana 


EWS is included in the state’s Every Student Succeeds Act plan, and all schools that are identified as comprehensive or 


targeted support schools will be required to use the Montana EWS. 


The system implementation process varies from school to school. The OPI is typically involved when schools make 
the initial decision to use the Montana EWS, and which OPI staff are involved is determined based on the needs of the 


specific school. Training is provided on-site at the school, and it addresses setting up an early warning system team, how 


to pull the data needed for the Montana EWS, how to read system reports, and other information. Training also includes 


a review of the school’s analyzed early warning data, which identify student risk levels. 


Montana does not have a statewide SIS that must be used by all schools across the state. This poses certain challenges 


in supporting statewide planning and implementation of the Montana EWS. The Montana EWS requires schools to 


upload data into the state data warehouse, and because schools use different SISs, system capability and data uploads can 


be challenging. To alleviate some of these issues, the OPI established a required format that must be used by all schools 


when uploading data into the Montana EWS. The OPI has also worked with schools and vendors to develop a data 


extract feature for the two most commonly used SISs in the state. 
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Early Warning Indicators, Data, and Analytics 


The OPI used both external research and internal data to identify early warning indicators. OPI staff consulted external 
research to determine which indicators might be helpful to include in the Montana EWS. The OPI then analyzed its 
internal data to determine whether these research-based indicators were valid dropout predictors for Montana students. 


The Montana EWS includes four locally validated indicators: attendance, behavior, grades, and mobility. 


No new data collections were required at the state or local level to Early Warning Indicators 


implement the system, as most of the data required to use the Montana 
EWS are already collected by schools. The OPI provides roughly half * Attendance 
e¢ Behavior 

¢ Grades 

e Mobility 


of the data required to use the Montana EWS from datasets already 
collected and reported by schools and districts. These data are stored in 
the state data warehouse. Individual schools must provide the remaining 


data. Most of these data are already collected by schools but are not Early Warning Indicator Data Elements 


reported to the state. The Montana EWS has a feature in place to 
account for data elements that are not collected by the school, which 
reduces the potential burden associated with implementing a new data 


collection to specifically support EWS implementation. 


The Montana EWS is stored in the state data warehouse. An overview 
of the Montana EWS is available to the public, but the Montana EWS 
results are only available through a secure, password-protected version 
of the site. Each user must submit authorization forms, which requires 
signatures from the proper school representatives, to receive a login and 
password for the secure version of the site. Individual schools decide 
which school staff are granted access to the system and OPI controls 


overall system access. 


Several OPI staff work year-round to ensure that data are entered 


correctly and in a timely manner. Data that are collected by the state are 


Attendance rate 

Absences in the last 60 days 
Absences in the last 90 days 

Number of behavior events in the last 
120 days 

Number of out of school suspension 
events in the last 3 years 

Credits earned per year 

Number of previous term A’s 

Number of previous term F’s 

On track status to graduation 

Age difference from peers* 
Previously dropped out status 
Repeating a grade in K-8 (Retention)* 
Mobility this school year* 

Mobility from out of state* 

Number of towns/districts the student 
has attended school in* 


updated on a nightly basis. Because the Montana EWS requires schools Other Data Elements 


to upload their own data, schools control the frequency, consistency, e Age 


and quality of their data. Data uploads from the schools are typically eeCenden: 


performed by data staff, while data analysis and reporting are usually 


*Data are collected by OPI 


completed by a curriculum director, counselor, principal, or other 
staff at the school. The quality of the data uploaded by the schools is 
the responsibility of the school, and the OPI has taken certain steps to 


help ensure schools report quality data. In addition to providing training to schools, the state data warehouse has several 
data quality checks to make sure data are formatted correctly. Data that are uploaded to the system must pass these data 
quality checks to be accepted; users receive an error message when data are not accepted. Most schools choose to update 
their data monthly, but the system’s use of current data enables schools to update system data on a daily basis if they so 


choose. 
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The Montana EWS uses a logistic regression analytical model to produce a percentage chance of dropping out based on 
each student’s current data. Students are assigned a level of risk based on their results: low risk, at risk, and extreme risk. 
These results are made available to schools immediately after the data upload process is complete. The system provides 
these results based on the student’s current data and stores the dated results from the previous 12 times that data have 
been uploaded for that student. The Montana EWS also generates school-level reports that aggregate the results of all 
the students in the school or district. In these reports, schools can see what percentage of students are flagged for each 
of the system’s four risk indicators: attendance, grades, behavior, and mobility. These data are available for all students in 
the school and at each grade level. The state average and the previous results for that school are also provided. Using this 
information, schools can see in which areas their students struggle when compared to students in other schools in the 
state. 


System Use 


Each school uses the system, its data, and results a bit differently based on their individual needs. At most schools, 
principals and counselors are the main users of the system’s early warning data. Other users may include curriculum 
directors, data analysts, teachers, and other school staff. At certain schools, students are provided with their individual 


results to help them better understand why they may be at risk of dropping out of school. 


Schools can use the Montana EWS’s reported results to identify which students at the school may need additional help, 
and then design targeted interventions that address why individual students, and the school overall, are struggling. Each 
school is responsible for developing and assigning students to intervention programs. The system’s three risk categories 
(low risk, at risk, and extreme risk) correspond with many of the tiered intervention programs used by schools in the 
state, which can help schools determine which students should receive which types of intervention (universal, targeted, 
or intensive intervention). To evaluate the effectiveness of interventions, schools can use the system’s longitudinal data to 
track students who are receiving interventions and determine whether those students benefited from the interventions 
they received. 


The OPI has made several changes to the Montana EWS since the agency first implemented the system during the 
2011-12 school year. Many of these changes were made based on feedback and suggestions provided by users from the 
field and include modifying the early warning indicators and moving the Montana EWS into the state data warehouse. 
The OPI also added new system features that allow schools to view different groups of students within the school and to 
see previous results for students that enroll in their school from another school within the state. The effectiveness of the 
system’s analytical model is re-evaluated annually during the summer. The OPI reviews the Montana EWS’s analytical 
model as well as feedback from users in the field, and then implements changes based on this evaluation to improve the 
model’s effectiveness. In addition to using the Montana EWS for evaluation purposes, OPI staff also use the Montana 
EWS to determine whether the schools they are working with upload data into the system, and if so, how often the 
system is used. 


Lessons Learned 


* Training is important. The OPI noted that providing appropriate training is helpful to schools that 
use an early warning system. The OPI provides training at opportune times, including prior to system 


implementation, and as schools start to use the Montana EWS. Training activities also incorporate early 
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warning data from the school to help school officials better understand the effectiveness of the system, how 
risk level is assigned, and how the school can use the system. Schools that fully understood the results of the 
Montana EWS’s results and how to use those results benefitted the most from the system. 

* Have a champion. OPI found that schools with at least one champion who understands and fully believes in 
the early warning system benefit the most from the Montana EWS. Having a champion in place can improve 
the likelihood that the school will continue to use the early warning system. 

* Encourage data use. The OPI observed that schools who regularly use the early warning system are more 
likely to fully benefit from the system and noted that creating an early warning system team that meets on at 
least a quarterly, and preferably monthly, basis to review the early warning data is helpful. Teams can review 
and discuss both school-level and student-level results, as well as how well current interventions are working 


for students who were identified as at risk of dropping out. 
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Case Study: Bozeman School District #7, MT 


The Montana EWS is a voluntary statewide early warning system Agency Background 
developed by the Montana Office of Public Instruction (OPI) in the 
2011-12 school year. The Montana EWS analyzes the risk level of 


students in grades 6-12 at participating schools to determine student 


Bozeman School District #7 serves 
approximately 6,500 students enrolled 

in 11 public schools. The adjusted 

risk of dropout. The Montana EWS is used by the OPI to measure cohort graduation rate is 84 percent. 
state dropout trends, and participation in the Montana EWS is at the Approximately 89 percent of the student 
population is categorized in the White 
racial/ethnic group, while the next largest 
student groups are the Hispanic/Latino 


discretion of individual schools and districts. The system is used in 
schools throughout Bozeman School District #7—including the district’s 
high school, Bozeman High School, and one of the district’s middle 


group (4 percent) and the Two or More 


schools, Chief Joseph Middle School—with the support of the district’s Races group (2 percent). Approximately 
central office. 10 percent of students have an IEP and 
1 percent are English learners. (Source: 
System Planning and Implementation ED, NCES, CCD, School Year 2015-16; 
EDFacts, School Year 2015-16). Note: 
Multiple factors motivated Bozeman School District #7 to consider These numbers represent the most 


implementing the Montana EWS. Staff and administrators were recent ED data and do not necessarily 
interested in more comprehensively and holistically examining student reflect the latest district data. 

needs, The district was also familiar with the Montana EWS through its 

participation in Graduation Matters Montana and relationship with REL 


Northwest, the Regional Educational Laboratory that promotes the use of Montana’s early warning system. 


The district formed a committee to consider the feasibility and appropriateness of implementing the Montana EWS. 

The District EWS Committee was comprised of building and district leaders (including the deputy superintendent, 
director of special education, high school principal, middle school principals, and elementary school principals), IT staff, 
central office staff, and Graduation Matters Montana representatives. Middle and high school principals also appointed 
appropriate staff from their schools to the committee, including assistant principals and at least one counselor from each 
building. Over the course of several years, the District EWS Committee coordinated its data systems, set a schedule for 
reporting frequency, established targets and goals for dropout prevention strategies across the district, and vetted student 


intervention options. 


Bozeman School District #7 joined Project Responsive Education for All Learners (REAL) during the early warning 
system planning process. Project REAL is a grant-funded multi-tier system of supports (MTSS). Bozeman School 
District #7 uses the Montana EWS as an extension of the district’s MTSS program and has used the Montana EWS 
consistently since the 2016-17 school year. To support planning and implementation at the school level, individual 
schools established teams to oversee the planning process and support the successful implementation and use of the 
Montana EWS. Communication with staff regarding the Montana EWS was important for creating an understanding 

of the system. The district also received extensive training and guidance prior to implementing the Montana EWS. The 
Bozeman School District #7 central office provided training before and during implementation. Additionally, the district 
participated in REL focus groups; organized meetings, workshops, and site visits; and consulted resources and technical 
assistance on early warning systems from the Montana Education Research Alliance, REL Northwest, and Graduation 
Matters Montana. 
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Early Warning Indicators, Data, and Analytics 


The Montana EWS uses four early warning indicators—attendance, behavior, grades, and mobility—to calculate each 


student’s dropout probability. The system also monitors good standing and changes in student performance and dropout 
probability. The district uses the same indicators as the OPI-created Montana EWS. 


Data on mobility, retention, and gender are collected by OPI and imported into the Montana EWS, while attendance, 


behavior, and course performance data are collected and uploaded by districts. These data were already available, and no 


new school, district, or state data collections were necessary to support the Montana EWS. 


The district uses several vendor-provided information systems. The District EWS Committee developed procedures 
for importing district data from these systems into the Montana EWS. At Chief Joseph Middle School, behavior and 


academic support staff, as well as MTSS leadership teams, are responsible for inputting, analyzing, and reporting early 


warning data to the district. At Bozeman High School, the high school’s data teams work with the early warning data. 


Each data team includes the school’s deans, a counselor, an administrator, and, as appropriate, support personnel such 


as a parent liaison, school psychologist, student support specialist, or 
nurse. The district data specialist is responsible for importing the early 
warning data into the Montana EWS for each participating school in 
the district. The district data specialist uses Structured Query Language 
(SQL) code to extract and upload the data necessary for the Montana 
EWS. The SQL code was developed by another Montana district 

that uses the same vendor-provided SIS. The query necessary for 

data extraction is complex and required considerable time to modify 
and refine. District data are pulled from the SIS and uploaded to the 
Montana EWS three times a year in fall (October), winter (January), 
and spring (May). The district data specialist also analyzes the data and 
prepares spreadsheets and reports to share with schools within the 
district. Principals, counselors, and MTSS teams are responsible for 


analyzing student-specific information at the school level. 


Numerous steps are taken to ensure data security and protect student 
privacy. All data collection and reporting systems are secure, and early 
warning data are uploaded to OPI over Secure File Transfer Protocol. 
The system does not yet include automated data checks, but each data 
extraction includes data verification steps. Reports and student lists are 
shared confidentially on a need-to-know basis with administrators and 
counselors who hold an appropriate security clearance. Furthermore, 
all meetings that include reviews of the early warning data are 
confidential. Regular reviews of the early warning data have identified 
opportunities to improve data quality. Bozeman High School staff, for 
example, noticed that a few students identified as at risk of dropping 
out seemed like they should not be on the list. Administration consulted 
with central office staff and worked to correct historical grade issues 
that had impacted the accuracy of the Montana EWS’s results. 
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Early Warning Indicators 


Attendance 
Behavior 
Grades 
Mobility 


Early Warning Indicator Data Elements 


Attendance rate 

Absences in the last 60 days 
Absences in the last 90 days 

Number of behavior events in the last 
120 days 

Number of out of school suspension 
events in the last 3 years 

Credits earned per year 

Number of previous term A’s 

Number of previous term F’s 

On track status to graduation 

Age difference from peers* 
Previously dropped out status 
Repeating a grade in K-8 (Retention)* 
Mobility this school year* 

Mobility from out of state* 

Number of towns/districts the student 
has attended school in* 


Other Data Elements 


Age 
Gender* 


*Data are collected by OPI 


41 


The Montana EWS uses multivariate logistic regression to calculate dropout probability and identify students at risk of 
dropping out. Students considered at risk of dropping out typically have one or two indicators that strongly influence 
their risk level. Support staff analyze each student’s specific results according to the student’s primary risk factor(s). 
Analysis of the model’s accuracy to predict dropout within the district occurred in the 2017-18 school year. In addition 
to identifying student risk level, the system flags changes in student results with an arrow to help readily identify changes 
or patterns in student risk level over time. Bozeman High School staff devote time during data meetings to focus on 
students at chronic risk, as well as students who have a change in either their EWS risk level status or at least one early 


warning indicator. 


System Use 


Implementing the Montana EWS has started important conversations within Bozeman School District #7. High- 

risk students are typically already known to support staff and administrators, and the Montana EWS data have helped 
identify students before they become high-risk, such as students who are receiving marginal, but not yet failing, grades. 
This supports early interventions, which are often more successful than interventions that occur late in the student’s 


educational career when the student may be at high risk of dropping out. 


Bozeman School District #7 uses multiple strategies to encourage early warning data use. The district data specialist 
prepares spreadsheets and reports for each school in the district that uses the Montana EWS. School-level teams receive 
these reports from the central office for use in their intervention programs. Principals, counselors, and MTSS teams 
are the primary users of the EWS results at the school level. At the district level, the central office data specialist and 
administrators of the School and Family Engagement — Trauma Informed grant use the Montana EWS data as one metric 


in determining district success in supporting students. 


At Bozeman High School, the data teams review Montana EWS reports and develop an intervention plan for each 
student identified as at-risk. Regularly scheduled meetings occur every 3-6 weeks and attendance is non-negotiable, 
meaning that no other activities on a team member’s schedule may take precedence over data team meetings. Typically, 
the data teams review the 30 students who demonstrate the highest level of risk by the Montana EWS, with special 
attention paid to students with credit deficiencies and students who are new to Bozeman High School. The high school 
has a list of interventions to help match students with interventions that will best meet the student’s needs. A designated 
member of each data team is responsible for coordinating with all appropriate personnel who will support the 
intervention plan. For example, if a student has attendance issues, staff will call the student’s parent, and the student’s 
attendance data will be shared with a dean for future monitoring. Administration will notify teachers if a student’s grades 
are a concern and when an intervention plan is in place to address academic issues. If a student who is new to the high 
school is at risk of dropping out, an intern calls their parent(s) or guardian(s) and a check-in postcard is mailed to the 


student’s home. 


At Chief Joseph Middle School, Montana EWS data are reviewed by MTSS leadership teams. These teams use key 

data elements to identify students at risk of dropping out, meet to review and discuss the progress of students and 
intervention programs, facilitate communication of goals and action steps for counselors and teachers, and define work 
tasks. Data are disseminated on a need-to-know basis and are also used by building administrators, student services 
staff, and counselors as a fidelity check of their schoolwide MTSS implementation. The middle school uses a three-tier 
MTSS to address academics and behavior as part of their participation in Project REAL 2.0, a Montana State Personnel 


Development Grant. Fidelity checks also include site visits and self-monitoring tools, such as rubrics. The results of the 
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fidelity checks are used to evaluate the effectiveness of the MTSS and develop action steps to support improved MTSS 
implementation, sustainability, and use. Students who are identified as at risk are referred for a tier II or III intervention 
provided within the MTSS framework. At the tier II level, Montana EWS data are used to create problem statements 
and develop plans to address the problem statement. The tier II team is part of the decisionmaking process. Tier II] is 

a more intensive approach to meeting the needs of at-risk students, and data drive the discussion and decisionmaking 
process. Intervention plans are developed by teams and/or the school psychologist using a menu of interventions; these 
interventions include programs, literacy workshops, and math strategies. The Montana EWS has guided a schoolwide 
effort to increase average daily attendance, as well as identify students at risk of dropping out due to attendance 


concerns. 


Now in its third year of implementation, the district conducted an internal validation of the system in the 2017-18 
school year; the analytical model is refined on an annual basis by the OPI. Longitudinal data will show if dramatic 
changes within the district’s student population have occurred over time, if annual seasonal patterns are impacting 
students, and if students experiencing significant life events or challenges with transitions are inhibited from success. 
Looking ahead, the district hopes to use what they learn from the Montana EWS to build a similar system to predict 
middle school success so that elementary school students can receive support and interventions early in their educational 
career. Additionally, there is a desire for the Montana EWS to be better integrated with the district’s SIS to review trends 
in individual student risk over time. 


Lessons Learned 


* Devote adequate time to planning. The Bozeman School District #7 central office allowed adequate 
time to verify the accuracy of the Montana EWS data and model before implementation. The additional time 
devoted to verification was worthwhile, even though the Montana EWS was not available as quickly as users 
would have preferred. Chief Joseph Middle School also recommended developing a plan for communicating 
with staff throughout the development and implementation process. 

* Customize approaches. The district found that a standardized approach may not be useful for schools or 
students. Bozeman High School found it challenging to assign students to one-size-fits-all interventions, and 
the central office noted that it was useful to find out what works best for individual schools. 

* Set clear team expectations. Both Bozeman High School and Chief Joseph Middle School benefitted 
from clear expectations for team members and team meetings. Chief Joseph Middle School developed MTSS 
leadership teams and assigned roles to members, including facilitator, timekeeper, data analyst, and recorder. 
The Bozeman High School data teams have set a clear building-wide expectation that data team meetings take 
priority over anything else on the schedule. Their regularly scheduled meetings apply a human perspective to 
the Montana EWS data, which helps add important contextual factors that may not be readily apparent in the 
data. 

* Use data to support teams. Both Bozeman High School and Chief Joseph Middle School found the 
Montana EWS data useful for ensuring students and their data are reviewed in a systematic way, and that data 
are useful for informing discussions, actions, and interventions. 

* Celebrate successes. The goal of the Montana EWS is to help students graduate from high school. Bozeman 
High School staff noted that it is rewarding when students who were at high risk of dropping out are able to 
get back on track and graduate from high school. Students who are doing well are sometimes recognized with 
a positive postcard or gift card. 
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Case Study: Metro Nashville Public Schools, TN 


Metro Nashville Public Schools (MNPS) created an early warning system 
in 2010 as part of a broader commitment to encourage data-informed 
school improvement across the district. MNPS was interested in adopting 
sustainable, proven strategies to improve student achievement and 
graduation rates, but was challenged by a high dropout rate and a high 
number of discipline incidents resulting in out-of-school suspensions and 
expulsions. In response to these challenges, the Tennessee Department of 
Education encouraged the district to systematically use data to identify 
district, school, and student needs; prioritize actions to improve; and 


provide targeted interventions to students in need. 


System Planning and Implementation 


The early warning system planning process began when the MNPS 
Department of Research, Assessment, and Evaluation partnered with 

the Everyone Graduates Center (EGC) at John Hopkins University 
(JHU). The Department and Center reviewed 10 years of MNPS data to 
identify locally valid early warning indicators and thresholds specific to 
the district. Predictive indicators of student risk of dropping out were 
identified for students in elementary, middle, and high school. Staff in the 


Agency Background 


Metro Nashville Public Schools serves 
approximately 86,000 students enrolled 
in 167 public schools. The adjusted 
cohort graduation rate is 81 percent. 
Approximately 42 percent of the district’s 
student population is categorized in the 
Black or African American racial/ethnic 
group, while the next largest student 
groups are the White group (30 percent) 
and the Hispanic/Latino group (22 
percent). Approximately 71 percent of 
students qualify for free or reduced lunch, 
12 percent have an IEP, and 15 percent 
are English learners. (Source: ED, NCES, 
CCD, School Year 2015-16; EDFacts, 
School Year 2015-16). Note: These 
numbers represent the most recent ED 
data and do not necessarily reflect the 
latest district data. 


Department of Student Support Services were also involved in reviewing how the early warning data would be translated 


into intervention programs and services for students flagged by the system as at risk of dropout. 


At that time, MNPS was implementing a new longitudinal district data warehouse system using Race to the Top 
grant funding. MNPS IT staff modified and extended a vendor-created technical platform to create the district’s data 


warehouse. These modifications supported the district’s need for additional data storage and advanced data analytics 


for use in data-informed decisionmaking. In 2011, the data warehouse team and IT staff collaborated with Research, 


Assessment, and Evaluation staff over 2-3 months to develop technical guidelines and requirements for an early warning 


system in the district data warehouse. 


District leadership was supportive of the early warning system and was also made aware of the goals of creating the 


system. MNPS used resources on early warning systems to ensure that development and use of the district early warning 


system was informed by best practices. However, there was no formal implementation plan, which slowed the adoption 


and use of the system districtwide. 


Early Warning Indicators, Data, and Analytics 


MNPS analyzed its internal longitudinal data and reviewed current research by early warning system researchers 


to identify and validate its early warning indicators. Research from the JHU EGC and the University of Chicago 


Consortium on School Research was particularly helpful during the indicator development process. The district’s early 


warning system uses three locally validated indicators: attendance, suspensions, and GPA. These indicators are aligned 


with the early warning “ABCs”: attendance, behavior, and course performance. 
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All early warning system data are collected as part of the district’s regular data collection activities; no new data 
collections were needed to support the early warning system. All data used in the early warning system are readily 
available in the district data warehouse and through extracts from the district’s SIS. The data warehouse houses the early 
warning system and its data, and delivers reports to school, district, and community stakeholders. System data are 
updated nightly with newly reported data, such as attendance and behavior data, which are recorded daily; grades are 
reported on a quarterly basis. Student records are protected through a custom security model. Access to data is limited 
by staff role: district staff can access information on all schools within 
the district; school staff can access information on students within their Early Warning Indicators 


school; and classroom teachers and staff can access information on 
Attendance 


Suspensions 
Grade point average 


students enrolled in the courses they teach. 


The district’s data collection and reporting processes are supported 
by multiple staff at the school and district levels. Each school is Early Warning Indicator Data Elements 


responsible for entering individual student data into the SIS; these 


Daily attendance, year-to-date 
data are imported into the district data warehouse nightly. IT staff are Number of days suspended 


responsible for maintaining and expanding the district data warehouse, Quarterly GPA (0-100) 


as well as supporting the development of data reports in response to 
stakeholder needs. Automated data checks are conducted nightly as the 
data warehouse receives individual student data from the SIS. The Department of Data Quality and Integrity monitors 
data accuracy and assists in resolving the root cause of quality issues as they arise. Data quality dashboards and reports are 


also made available to school and district staff to help staff monitor, identify, and correct errors that impact data quality. 


The early warning system uses a multivariate analytical model to assess student risk of dropout; data on each indicator 
are also reviewed independently. Each indicator has a predefined threshold that flags a student as at-risk. Risk levels 

are automatically calculated nightly using the most current data available, which enables staff to identify students in 
need of intervention as soon as possible. Students that are flagged as at-risk are highlighted on data dashboards to help 
prioritize action and interventions. The Department of Research, Assessment, and Evaluation is responsible for ensuring 
analytical model validity and providing information and guidance on algorithms and analysis methods. Data warehouse 
development/IT staff create accurate and meaningful data reports with features that can help stakeholders act on the 
data. For example, data drill-downs within a dashboard feature of the system enable users to review lists of individual 
students. 


Early warning data reports are often used in conjunction with aggregate datasets and individual student data reports. For 
example, data on the early warning attendance indicator are disaggregated, interpreted, and used in tandem with other 
attendance reports that are disaggregated across different factors. Interpreting early warning data in combination with 
related datasets enriches understanding of the data and can prompt more effective interventions at the student or school 
level. Major changes in individual student performance that do not flag a student as at risk of dropping out, such as a 
major decline in assessment performance, are monitored through separate reports and data systems. Positive or negative 
growth trends across various student achievement measures are monitored through the data warehouse, which reports 


trends based on different units of time: a reporting quarter, a single school year, or multiple school years. 
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System Use 


The extent to which the system is used districtwide differs between individual schools and departments. While no 
formal communication plan exists, clear communication and expectations are critical to supporting districtwide use. 
MNPS communicates information about the early warning system through multiple channels. Information about the 
early warning system and its indicators are shared with stakeholders at all levels—classroom, school, district, and school 
board—as well as a limited number of afterschool and community-based programs. Training and system demonstration 
sessions are provided during team meetings and professional development activities. Training information is tailored 

to the specific audience’s initiatives and work processes to help make the system more relevant to specific stakeholder 


groups. District data coaches and school intervention staff also share information on the system with the teams and staff 


with which they work. 


The district encourages early warning data use in a variety of ways. Rather than existing as a standalone tool or dataset, 
the early warning system and its data are a key component of MNPS’s districtwide commitment to data use. Data 

use is embedded in the MNPS strategic plan and promoted through district- and school-level meetings with staff and 
leadership. School leaders, data champions, and staff from data-focused departments serve as role models for data use 
within their school and promote the early warning system as a tool that can help solve problems that district leadership 
care about. 


The Department of Student Support Services was an early adopter of the early warning system. Support teams—which 
include behavior specialists, attendance specialists, family engagement specialists, social workers, psychologists, and 
others—use the early warning system to manage staff caseloads, identify students in need of intervention, and develop 
intensive intervention plans. The system helps staff identify students who are at highest risk and prioritize these students 
for intervention services to ensure that department resources are allocated to supporting students who are most in need 


of interventions. 


The early warning system is also used by staff and administrators at the district and school levels. At the district level, the 
system helps district leaders understand the extent of student dropout risk across the district. If the system identifies a 
high number of students as at risk across multiple indicators, district leaders work with individual schools to understand 
why students are at-risk. At the school level, counselors, staff, and leadership use the system to identify which students 
in their school are least likely to graduate. Teachers often have a strong understanding of individual student needs and 


circumstances, which can be helpful when referring at-risk students for specific intervention services. 


MNPS has a variety of intervention programs and services to support students who are at risk of dropping out. 

Families of students who exhibit risk across multiple indicators are now more likely to receive a home visit from family 
engagement and intervention staff. Many schools have formal partnerships with community-based agencies that work 
with students on a referral basis. Additionally, many schools have dedicated “intervention periods” during the school day 


to provide students with services or content that meet individual student needs. 


The district’s overall commitment to improving data use has supported sustained, incremental improvements to acting 


with education data at all levels. Maintaining system flexibility and using and leveraging existing resources has enabled 
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MNPS to adapt to changes, make improvements, and meet stakeholder needs in innovative ways. The early warning 
system has contributed to this effort by improving the district’s ability to identify and provide interventions to students 
who exhibit risk of dropping out of school, which has improved graduation rates over time. Since the system was first 
implemented, the Department of Research, Assessment, and Evaluation has twice evaluated the effectiveness of the early 
warning system to confirm the predictive value of the early warning indicators. The district is currently conducting an 
analysis of more recent data with JHU EGC to determine cut scores for the off-track (dropout) indicator and a new on- 
track (college and career ready) indicator. The district plans to analyze National Student Clearinghouse data to validate 
college readiness indicators. The district also hopes to analyze de-identified data from the state’s longitudinal data system 
that are not currently available to the district, such as labor participation data and data from Tennessee Colleges of 


Applied Technology, a state-supported college system with 27 campuses. 


Lessons Learned 


* Invest in adoption. MNPS began developing its early warning system in 2010. At that time, early warning 
systems were not as well-known and widely used as they are today. As an early adopter, the district had 
minimal external encouragement and resources to support system development and use. MNPS noted that 
establishing an understanding of the system’s purpose and value amongst staff and engaging leadership in 
supporting system implementation and use might have encouraged broader and timelier adoption of the 
system in schools across the district. 

* Implementation is an ongoing process. MNP%S’s use of early warning data has advanced as changes occur 
in district staffing, priorities, and programming. MNPS noted that implementation is never complete, as the 
people and methods for using the early warning system change as the district changes. Identifying ongoing 
opportunities to leverage all data, including early warning system data, to inform stakeholders is part of this 
ongoing implementation process. 

* Encourage teacher use. Teachers are not required to review or act upon the early warning data. The 
district found that it can be helpful to provide resources that encourage teacher use, such as a toolkit to assist 
with selecting interventions or communication materials that explain the utility of early warning indicators. 
The district’s instructional coaches also encourage teacher data use by assisting teachers in identifying and 
accessing data that are useful and relevant to their area of focus. 

¢ Align programs with priorities. Early warning indicators can be used to create awareness for and align 
community programs with district priorities to ensure that community-based organizations provide programs 
and services that address student and school needs. The Nashville After Zone Alliance (NAZA), for example, 
evaluated their impact on student behaviors that are monitored through the early warning system. This 
evaluation found that participation in NAZA contributed to improved student attendance and discipline, both 
of which reduce student risk of dropout. 
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Case Study: Wisconsin Department of Public Instruction 


The Wisconsin Department of Public Instruction (DPI) created the 
Dropout Early Warning System (DEWS) during the 2012-13 school 
year as part of the State Superintendent’s Every Child a Graduate 
initiative. DEWS is a statewide early warning system that identifies 
students in grades 6-9 who are at risk of not graduating from high school 
on time (e.g., within four years from the start of grade 9). DPI’s goal for 
DEWS is to support schools as they leverage data to improve student 


educational outcomes. 
DEWS was developed to address five key goals: 


* Early — The system identifies whether students need 
intervention early in their educational career. 

* — Accurate — The system accurately identifies whether students 
need intervention. 

* Transparent — Predictions and the assignment of risk status are 
made in a transparent manner. 

* — Reproducible — Predictions are reproducible and do not vary 
based on changes in the prediction models. 

* — Scalable — System scalability accommodates a diverse array of 


student and school contexts. 


System Planning and Implementation 


Agency Background 


The state of Wisconsin serves 
approximately 868,000 students enrolled 
in 2,291 public schools within 472 school 
districts. The adjusted cohort graduation 
rate is 88 percent. Seventy-one percent 
of the student population is categorized 
in the White racial/ethnic group, while 
the next largest student groups are the 
Hispanic/Latino group (11 percent) and 
the Black or African American group (9 
percent). Approximately 39 percent of 
students qualify for free or reduced lunch, 
14 percent have an IEP, and 5 percent 
are English learners. (Source: Source: ED, 
NCES, CCD, School Year 2015-16; ED, 
OESE, School Year 2014-15). Note: These 
numbers represent the most recent ED 
data and do not necessarily reflect the 
latest district data. 


DPI research staff initiated the DEWS planning process in early 2012 through the exploration and prototyping of a 


predictive analytic model. During this process, DPI researchers partnered with the Wisconsin Educational Research 


Advisory Council—a council of researchers that advises the state superintendent on education research matters—and 


consulted research literature on early warning systems and predicting high school dropout. 


After a viable analytical model was developed, DPI formed a working group to develop and implement the system. 


The working group included staff and subject matter experts from across the SEA, including DPI’s Policy and Budget 


team, Student Services/Prevention and Wellness team, Data Warehouse team, Special Education team, Office of 


Educational Accountability, and others. A research analyst on the Policy and Budget team led the system’s technical 


and predictive model development and implementation process. The Student Services/ Prevention and Wellness team 


developed training materials and support documentation that address how LEAs can use DEWS as part of their student 


intervention programs and systems. 


DPI developed DEWS using open source software. The code is available for users and stakeholders to examine how 


risk scores are calculated, as well as for other education agencies to use in developing their own early warning system. 


During the system development process, the working group conducted a pilot test to assess DEWS prior to launching 


the system statewide. Select middle schools across the state participated in the pilot during spring 2013. DPI surveyed 


participating schools on the usefulness of the system and report formats, how well the system identified students’ risk 
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levels, the value of the interpretive guide and other supporting materials, and the school’s likelihood of future use. The 


survey feedback was used to refine the system and the training support materials prior to statewide implementation. 


The working group also created a communication plan, standard presentation materials, and a suite of training tools to 
publicize the availability of the new system to LEAs. During the summer before DEWS was launched, DPI staff delivered 
presentations at professional meetings, data retreats, and conferences throughout the state. These presentations reached 
several stakeholder groups, including principals, school counselors, special education coordinators, Title | coordinators, 
regional service agencies, and statewide RTI and Positive Behavioral Interventions and Supports (PBIS) networks. 


In addition to technical information and interpretive/action guides, DPI emphasized that DEWS was not part of the 
state’s accountability system and that risk assessments were not permanent labels, but rather intended to focus action 
on those students who are most in need of intervention. To this end, DPI decided to include DEWS scores only on 

the secure data dashboard system and only at the student level. This decision meant that some of the richer reporting 
features from the pilot had to be excluded. These user design decisions have impacted the adoption and usage of DEWS 
by LEAs. 


Broad collaboration across the agency was critical to ensuring that the development process produced a system that is 
accurate, understandable, and useful for practitioners at the district and school levels. Through the involvement of many 
DPI staff and a substantial commitment from DPI leadership, DEWS was launched statewide in fall of the 2013-14 


school year. 


Early Warning Indicators, Data, and Analytics 


DPI decided to include specific early warning indicators based on the availability of data in Wisconsin’s statewide 
longitudinal data system, internal preliminary analyses of indicator accuracy, and consultation with the research 
literature on early warning systems. DEWS uses multiple indicators, including attendance, discipline, mobility, test 
scores, and demographics. These early warning indicators are used to identify students who are at risk of not graduating 


from high school within four years. 


The Wisconsin statewide longitudinal data system is the source for all DEWS data, which influences the inclusion and 
exclusion of certain indicators and data that are commonly used in district- and school-level early warning systems. 
For example, course-level data are often included in early warning systems, but since these data are only available for a 
few years and not of high quality at the state level, they are not included in DEWS. Additionally, all test scores are from 
statewide accountability assessments, and discipline data are limited to out-of-school suspensions and expulsion and do 
not include less severe actions, such as in-school suspensions or detentions. DEWS also includes grade-specific school- 
level characteristic data in lieu of more direct measures of school climate to provide contextual information about the 


schools that may impact individual student outcomes. 


DEWS predictions are calculated based on a statistical model of the associations between on-time graduation and all the 
individual and school-level indicators for all available cohorts of students who are eligible to have completed (or not) 
four years of high school. Each year, separate predictive models are produced for each grade level by combining results 
from 3 to 8 different machine learning algorithms. For example, a current sixth-grader is assigned predictions based on 
the observed on-time graduation status of previous cohorts of fifth-graders whose observable characteristics are similar 


to their own observable characteristics from fifth grade. 
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The use of longitudinal data and grade-specific models maximizes the accuracy of student risk predictions and helps 


overcome some of the inherent limitations associated with using data from a single school year per student. DPI re- 


estimates new predictive models each year to ensure that risk score 
calculations reflect the latest information available in the data. At 
present, risk scores are calculated for more than 90 percent of all 
current sixth- through ninth-grade students using data from students’ 
fifth through eighth grade school years. DPI is currently investigating 
methods for generating risk scores for students who are missing data 


associated with one or more indicators. 


Risk scores are calculated twice each school year in alignment with 
when DPI receives data for the various indicators: once at the beginning 
of the school year and once toward the beginning of the spring 
semester as a new year of data become available. Risk scores may be 
calculated more frequently in the future as data collections within the 
state become more automated and real-time. A Policy and Budget 
team research analyst is responsible for running the model to update 
risk scores, and the Data Warehouse team’s IT staff are responsible for 
loading the updated scores into the state education data warehouse and 
secure dashboard system. Automatic reports are integrated into the 
risk estimation process to check data and model validity. Risk scores 
go through an additional quality check as they are loaded into the data 
warehouse. Any errors or issues are addressed before risk scores are 
posted to the state’s secure dashboard system for use by LEAs. LEAs 
may access risk scores for enrolled students through the dashboard, as 
well as the risk scores of students who move into their district from 
another district within the state. 


System Use 


Early Warning Indicators 


Attendance 

Discipline 

Mobility 

Test scores 

Demographics and other indicators 


Early Warning Indicator Data Elements 


Attendance rate 

Total days enrolled 

Number of discipline incidents 
Discipline incident type 
Removal type 

Total days removed 

Number of districts attended 
Number of schools attended 
Number of enrollment spells 
English/Language Arts test scores 
Mathematics test scores 

Full academic year indicators 
Disability status 

Economic disadvantage status 
Ever retained-in-grade 

English Learner status 

Gender 

Race-ethnicity 


DPI offers a variety of supports to encourage LEA adoption and use of DEWS. Training materials were 


initially developed by the DPI working group during pilot testing and are available on the DEWS website 


(https: //dpi.wi.gov/ews/dropout). Through a partnership between DPI and the state’s network of cooperative 


educational service agencies, the WISExplore project team provides training to LEAs on how to use data to inform 


decisions within a cyclical continuous improvement framework, including how to use and interpret DEWS scores. 


DEWS is available for use by all LEAs in the state, but not all LEAs use the system. Many of the largest districts in the 
state developed their own early warning systems, while other districts use the early warning system that is included as 
part of their vendor-provided SIS. Many of the smaller districts within the state can readily identify those students who 
are at risk of not graduating in four years, which also limits the use of DEWS throughout the state. 


DPI has taken several steps to increase the use of DEWS scores. A DEWS Action Guide is available on the DPI website, 


along with other links to specific supports, resources, and interventions aimed at helping students who are at risk of 


dropping out of high school. The WISExplore project team provides outreach to support DEWS use, often as part of 
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an RTI or PBIS system. DPI monitors usage of the secure dashboard system and sends announcements to LEAs when 
updated predictions scores are loaded to the dashboard. DPI also redesigned the dashboard to make it easier to find 
DEWS predictions and collaborated with REL Midwest to conduct a survey of DEWS awareness and usage in the fall of 
2015, 2 years after DEWS was launched statewide. 


DEWS is intended to be used by middle and high school principals, as well as school staff who are included in the 
student intervention process. DEWS risk scores are often used during monthly planning meetings or as part of planning 
meetings held at the beginning of the school year. These scores help teachers and administrators identify which students 
are at risk of not completing high school on time and assign those students to specific interventions based on student 
need. Coupling state-validated predictions with local knowledge of individual student circumstances can help schools 


provide interventions that will work best for their students. 


Since its statewide launch in fall 2013, DEWS scores have been loaded twice per year. DPI periodically receives user 
feedback and has implemented system updates, improvements, and changes in response. DPI continues to maintain, 
refine, and improve the system. Looking ahead, DPI plans to use scores from previous cohorts who have had time to 
graduate within four years to evaluate the accuracy of DEWS predictions and determine whether DEWS has influenced 


on-time graduation rates. 


Lessons Learned 


* Establish agency support. Broad support from DPI has played an essential role in the success of DEWS. 
DPI enlisted SEA staff and leaders from across the agency to share information on DEWS with LEAs. 
Additionally, the WISExplore team is an invaluable resource for training LEAs across the state on how to 
interpret and act based on DEWS risk scores. 

* Be transparent. DEWS uses a complex system of grade-specific predictive models. Rather than promoting 
DEWS as a one-stop solution or flawless system, DPI has found that clearly and objectively explaining the 
capabilities, limitations, and accuracy of DEWS and its predictions helps encourage LEA trust in the system. 

* In-house system development and maintenance is beneficial for end users. DPI staff developed 
and maintain DEWS in-house, which has made it easier for DPI to refine the system, update its prediction 
algorithms, and modify how risk scores are disseminated to users. This flexibility has also allowed DPI to 
respond to feedback from LEAs. DPI identified and addressed system issues from pilot users’ feedback before 
the system went live statewide. Additionally, a new college and career readiness early warning system was 
produced during the 2017-18 school year in response to user feedback. 

* Support each step of the early warning system process. As an SEA in a local control state, DPI 
and the WISExplore team are removed from direct interaction with students, cannot compel LEAs to adopt 
intervention programs, and are unable to develop an early warning system with integrated interventions and 
intervention monitoring. While these limitations can be challenging, DPI is committed to promoting and 
building capacity in Wisconsin LEAs as they focus on the data inquiry process and continuous improvement 


planning. 
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Conclusion: Lessons Learned From Case Studies 


This chapter presented seven case studies that detail the actual experiences of SEAs and LEAs that have implemented, 


or are in the process of implementing, an early warning system. A summary of the lessons learned from these SEAs 


and LEAs concludes this chapter, with the goal of helping agencies who are planning to implement, or have already 


implemented, an early warning system. 


System Planning and Implementation 


Secure broad support from staff and leaders across the agency by establishing a common understanding of the 
early warning system’s purpose and value. 

Draw on professional learning communities and the experiences of colleagues to learn about the barriers, 
strategies, and successes other states and districts have faced in developing, implementing, and using an early 
warning system. 

Involve individuals from diverse professional backgrounds in planning teams to ensure that early warning data 
are easy-to-use in analysis, evaluation, and decisionmaking. 

Collaborate with trusted partners to support the system development process and reduce development time 
and costs. 

Pilot test the system and solicit feedback to ease the system implementation process. 

Consider in-house system development and maintenance, which may make it easier to refine and improve the 
system based on user feedback. 

Develop a plan to communicate with staff throughout the system development and implementation process. 
Provide professional development to create awareness and engage stakeholders. 

Share information about the system prior to implementation to help build anticipation amongst stakeholders. 
Think of implementation as an ongoing process, not a one-time activity. 


Prepare for the system to evolve as changes occur in agency staffing, priorities, and programming. 


Early Warning Indicators, Data, and Analytics 


Budget a sufficient amount of time to develop the system’s analytical model. 

Balance analytical model complexity and transparency. 

Verify the accuracy of the early warning analytical model and data prior to implementation. 

Clearly and objectively explain to users the capabilities, limitations, and accuracy of the analytical model and 


its predictive abilities. 


System Use 
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Promote and build capacity for data inquiry and continuous improvement planning. 

Provide appropriate training at opportune times to help early warning system users understand the 
effectiveness of the system, how risk level is assigned, and how the system can be used across the organization. 
Create resources to encourage use, such as a toolkit to assist with selecting interventions or communication 
materials that explain the utility of early warning indicators. 

Identify ongoing opportunities to leverage all data, including early warning system data, to inform 


stakeholders. 


Use early warning data to inform discussions, actions, and interventions at the school, district, and state levels. 
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Establish an early warning system team to systematically review school-level and student-level early warning 
data and assess how well current interventions are working for students identified as at risk of dropping out. 
Prioritize regularly scheduled early warning system team meetings and develop responsibilities for each team 
member. 

Have an early warning system champion in place to help improve the likelihood that the system will continue 
to be used. 

Support each step of the early warning process by integrating early warning system use with intervention 
provision, assignment, and monitoring. 

Find out what works best for individual students and schools, and offer customized intervention programs and 
services that will best meet their needs. 

Work with community partners to align out-of-school interventions with education agency priorities to 
ensure that community-based programs and services address student and school needs. 

Celebrate student successes, such as recognizing formerly at-risk students when they graduate from high 
school. 
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Appendix A: Checklist of Early Warning System Tasks and Activities 


There are many operational tasks necessary to effectively plan, implement, and use an early warning system. The 
following list of activities can assist state and local education agencies (SEAs and LEAs) as they implement a new or 
improve an existing early warning system. This list is not exhaustive or prescriptive, and agencies should modify the tasks 


and activities in this process to best meet their needs. 


System Planning Tasks and Activities 
Form a planning team 


Y Include agency representatives from multiple stakeholder groups. 
v Develop roles and responsibilities for individual team members. 


Y Determine the purpose, scope, and focus of the system. 


Conduct planning activities 


Y Conduct a needs assessment to identify agency and stakeholder needs. 

v Engage stakeholders through focus groups, inter views, or public presentations and meetings. 

Y Define valid early warning indicators, identify high-quality data, and develop appropriate analytical techniques 
and procedures. 

v Develop goals and objectives for system planning, implementation, and use. 


Y Decide whether agency staff or an external vendor will develop the system. 


Identify indicators and data 


Review research on early warning indicators. 
Analyze internal longitudinal data for predictive trends and patterns. 
Identify potential early warning indicators. 


Select available, valid, and reliable data for each indicator. 


hs 


Validate early warning indicators and develop associated risk thresholds. 


Develop an analytical model 
v Design the analytical model. 
v Validate the accuracy of the model’s predictive abilities. 


Y Document the model’s analytical techniques and analysis procedures. 


System Implementation Tasks and Activities 
Allocate resources 


v Develop a budget that includes a cost-benefit analysis and a projection of staff time necessary for system 
development, implementation, and maintenance. 

v Assign responsibility for system development/procurement, implementation, and maintenance to appropriate 
departments and staff. 
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Develop/refine an early warning system 


Y Create a system development plan or request for proposal that includes goals, timelines, and deliverables. 
Y Document system requirements and create detailed technical specifications. 

v Establish system security procedures. 

v Align the system with other information systems. 


VY Pilot test system quality and usability prior to implementation. 


Train staff 


VY Create customized training materials to meet agency-specific needs. 

Vv Provide targeted training to specific departments and/or staff. 

Y Schedule staff training before, during, and/or after system implementation. 

v Incorporate the early warning system into training materials and activities on related topics. 
v Ensure that staff protect student privacy and abide by all data privacy regulations. 


Collect/import data 


Abide by all data privacy regulations. 
Assign data collection/import responsibilities to appropriate departments and staff. 
Enable automated data imports and quality checks. 


Se 


Establish a systematic data import, analysis, and reporting schedule. 


Analyze and interpret data 


Test data accuracy and quality prior to analysis. 
Analyze data using the system’s analytical model. 
Interpret system data with related student data. 


Sas NSN 


Implement data drill-downs to enable student-level data reviews. 


System Use Tasks and Activities 
Report and share data 


Ensure that data reports protect student privacy. 

Create a communication plan to guide data reporting and sharing activities. 

Assign responsibility for data reports to appropriate departments and staff. 

Make early warning data accessible through dashboards, reports, or other data outputs. 


I A ye 


Disseminate data to specific departments and staff who use early warning data. 


Provide intervention programs and services 


Encourage data-informed discussions and the systematic use of early warning data. 
Identify opportunities for the system to support current practices, programs, and processes. 
Promote data use through existing team-based structures and collaborative activities. 


Align the system with preexisting intervention systems, programs, and services. 


RAS 


staff. 
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Assign responsibility for intervention development, assignment, and provision to appropriate departments and 


55 


Monitor student progress 


Y Monitor students who are receiving interventions. 


v 


Monitor students who are no longer at risk of dropping out to determine the effectiveness of specific 
interventions. 


v Evaluate intervention program outcomes. 


System Evaluation Tasks and Activities 
Evaluate system accuracy and usefulness 


Y Evaluate the need for system upgrades during routine system maintenance. 
Y Evaluate the system’s accuracy in identifying students in need of intervention. 
Y Evaluate the system’s usefulness in helping staff and stakeholders identify students in need of intervention. 


Document stakeholder feedback 
v 


Develop formal or informal methods of soliciting stakeholder feedback, such as surveys, meetings, or other 
methods. 

VY Solicit stakeholder feedback on the system’s usefulness and actionability. 

v 


Review user statistics, such as login data, to determine the frequency and consistency of system use. 


Determine needed improvements 


v Identify needed system improvements. 


v 


Coordinate with agency staff and/or external partners to determine the feasibility and scope of system 
changes. 


v Develop a plan for implementing system changes. 
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Appendix B: Select List of Early Warning Indicators and Data 


Elements 


This appendix provides a list of early warning indicator topics that are commonly used in early warning systems, 


cross-walked with student-level indicators and related data elements. The indicators listed in table 2 are presented in 


alphabetical order, not in order of usefulness or power. In some cases, the early warning data would need to be derived 


or aggregated from the associated data elements. 


Many of these elements are not unique to early warning systems, and they may already be in use in existing data systems. 


While this list includes elements that are used in early warning systems, it does not include a comprehensive list of all 


possible data elements that could be included in an early warning system. State and local education agencies (SEAs and 


LEAs) may find this list useful as they build new data systems or expand existing systems to capture data related to early 


warning indicators. 


| Fate bter- troy me Keyes le 
Assessment 
performance 
Achievement or 


performance on 


an educational 
assessment, typically 
a test 


Attendance 
Attendance in an 
instructional program 
approved by the 
state, district, and/or 
school 


Behavior 

Negative or 
inappropriate 
behavior that results 
in a disciplinary 
action 


yap retcroums heXenternveyacy 


Completion of specific assessment(s), such as 
a college entrance exam 

Performance on state assessments and college 
entrance exams as compared to a specific 
target 


Attendance rates, including daily attendance 
and course attendance, as measured against 
chronic absenteeism thresholds that are 
aligned with school, district, and/or state 
thresholds 


The number and frequency of discipline 
incidents 
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Related Data Elements 


National percentile rank on grade-level 
assessments 

Performance on formative assessments or 
diagnostic tests 

Performance on school, district, and/or state 
assessments 

Performance on domain-specific and/or 
standards-based assessments 

Scores on diagnostic assessments 

Absences in the previous 60 days 

Absences in the previous 90 days 

Aggregate daily attendance 

Agetegate daily membership 

Attendance rate 

Chronic absenteeism 

Total days enrolled 

Discipline incident type 

Number of behavior events in the previous 

120 days 

Number of discipline incidents 

Number of discipline referrals 

Number of suspensions (in-school and out-of- 

school) 

Number of in-school suspensions 

Number of out-of-school suspensions 

Removal type 

Total days removed 
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| Fate bter- troy me Keyes le 


Course 
performance 
Academic 
achievement or 
performance across 
all courses or a select 
set of courses 


Mobility 
Geogtaphic 
movement between 
states, districts, and/ 
or schools 


Progression 
On-time 
advancement to the 
next level or grade 
in an instructional 
program 


yap retcroum bekenterineyucy 


Grade point average 
Number of courses with grades = F 


Geographic movement between states, 
districts, and/or schools 
Number of times a student changes schools 


Courses completed toward meeting diploma 
requirements or courses needed/pending 
(used to help measure whether the student 
is on track to graduate) 

Over-age status (is the student older than the 
average age for his or her grade?) 

Percentage of total needed credits earned to 
date 

Retention (whether a student has been retained 
in a grade or not) 


Table 2. Select list of early warning indicators and data elements 


Related Data Elements 


Common course code, section, year taken, and 
teacher of record 

Course grades 

Course failures 

Credits earned per year 

Grade point average 

Number of previous term A’s 

Number of previous term F’s 

Participation in accelerated learning programs, 
such as Advanced Placement (AP), dual 
enrollment, International Baccalaureate 
(IB), honors, etc. 

Participation in basic programs, such as 
English learners, gifted and talented, special 
education, etc. 

Grade of entry 

Military connectedness 

Mobility during current school year 

Mobility from out of state 

Number of districts attended 

Number of enrollment spells 

Number of schools attended 

Transient status 


Accrued high school credits 
Age difference from peers 
Cohort graduation year 

Ever retained in grade 

Number of years in high school 
On-track status to graduation 
Repeating a grade in K-8 
Retained designation 


In addition to data related to specific early warning indicators, early warning systems may include the following types of 


general student data: 


* Identification data, including name and student identifier 


* School enrollment data, including LEA and school identifiers, as well as entry and exit dates 
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Additional Resources 


Diplomas Now 

http: //diplomasnow.org/ 

Diplomas Now, funded in part by a U.S. Department of Education Investing in Innovation Fund (i3) grant, is a program 
that helps middle and high schools develop a comprehensive plan to improve student achievement. Diplomas Now 
partners with schools to set goals, develop a strategic plan, implement an early warning system, and engage in regular 
data reviews. The program also provides curriculum, teacher coaching, and student support to help each student get 
back on track. 


Early Warning Systems in Education 


http: //www.earlywarningsystems.org/ 


American Institutes for Research (AIR) has developed action-oriented, research-based resources and tools that states, 
districts, and schools can use to identify students in need of targeted interventions and supports to help student 
achievement and success. AIR offers tools, resources, technical assistance, and training to support the design and 


implementation of early warning systems. 


Early Warning Systems Learning Series 


https: / /ies.ed.gov/ncee/edlabs/projects/ews.asp 

The Institute of Education Sciences’ Regional Educational Laboratory (REL) Program hosted a learning series on 
implementing and using early warning systems for dropout prevention. The projects in the series include a variety 
of technical assistance activities related to developing and implementing early warning systems. The RELs produced 
a variety of resources and products on early warning systems, including webinars, in-person events, videos, tools, 


infographics, reports, and works in progress. 
graphics, reports, prog 


Everyone Graduates Center 

http: //new.every1graduates.org/ 

The Everyone Graduates Center, a unit of the Johns Hopkins University School of Education, seeks to identify the 
barriers that stand in the way of all students graduating from high school prepared for adult success, develop strategic 
solutions to overcome these barriers, and build local capacity to implement and sustain these strategic solutions. Their 
website features multiple resources on the effective development, implementation, and use of early warning systems, 


including compilations of resources used by state partners. 


U.S. Department of Education, Office of the Chief Privacy Officer 

https: / /studentprivacy.ed.gov/ 

The U.S. Department of Education’s Office of the Chief Privacy Officer (OCPO) is responsible for the administration 
and enforcement of federal laws relating to the privacy of students’ education records, and for the provision of technical 
assistance on student privacy issues for the broader education community. Guidance and resources on protecting student 


privacy, including publications, training materials, and technical assistance, are available through the OCPO website. 
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What Works Clearinghouse 
https: / /ies.ed.gov/ncee/wwe 
The What Works Clearinghouse (WWC), an investment of the Institute of Education Sciences (IES) within the U.S. 


Department of Education, reviews the existing research on education programs, products, practices, and policies, with 


the goal of providing educators with the information they need to make evidence-based decisions. The WWC includes 


resources on dropout prevention to help students stay in, progress through, and complete high school. 


National Forum on Education Statistics Resources 


Forum Guide to Building a Culture of Quality Data: A School and District Resource 
(2004) 

https: / /nces.ed.gov/forum/pub_2005801.asp 

This guide shows how quality data can be achieved in a school or district through the collaborative 


efforts of all staff and offers recommendations to staff in schools and school districts about best 


practices for improving data quality. 


Forum Guide to Collecting and Using Attendance Data (2018) 
https: / /nces.ed.gov/forum/pub_2017007.asp 


This guide provides best practice suggestions, real-life examples, a standard set of attendance codes, 


Fo 
Collecting and Using 
Attendance Data 


and role-specific tip sheets to help state and local education agency staff improve their attendance data 


practices. 


Forum Guide to College and Career Ready Data (2015) 

https: / /nces.ed.gov/forum/pub_2015157.asp 

This guide outlines the data needs and helpful analytics for five use cases (individual learning plans, 
educator support systems, postsecondary feedback loops, accountability systems, and career technical 


and education programs) that support SEA and LEA college and career ready initiatives. 


Forum Guide to Data Visualization: A Resource for Education Agencies (2016) 
https: / /nces.ed.gov/forum/pub_2017016.asp 


This guide recommends data visualization practices that will help education agencies communicate 


data meaning in visual formats that are accessible, accurate, and actionable for a wide range of 
education stakeholders. 


Forum Guide to Education Data Privacy (2016) 
https: //nces.ed.gov/forum/pub_2016096.asp 


This guide was developed as a resource for state and local education agencies (SEAs and LEAs) 


to use in assisting school staff in protecting the confidentiality of student data in instructional and 
administrative practices. SEAs and LEAs may also find the guide useful in developing privacy programs 


and related professional development programs. 
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Forum Guide to Education Indicators (2005) 
https: //nces.ed.gov/forum/pub_2005802.asp 


This guide provides encyclopedia-type entries for 44 commonly used education indicators. Each 


indicator entry contains a definition, recommended uses, usage caveats and cautions, related policy 
questions, data element components, a formula, commonly reported subgroups, and display 
suggestions. 


Forum Guide to Taking Action with Education Data (2012) 

https: / /nces.ed.gov/forum/pub_2013801.asp 

This guide provides practical information about the knowledge, skills, and abilities needed to identify, 
access, interpret, and use data to improve instruction in classrooms and the operation of schools, local 


education agencies, and state education agencies. 
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